In the Spread of Disease Model in the Models Library, when I run the BehaviorSpace experiment ‘population-density’ and select Table and Stats Output using the default settings involving 10 repetitions, I do not get mean and standard deviation for number of ticks taken (for full infection) calculated across repetitions in the output stats file. Instead, every repetition is treated as an independent run, the mean is given as the ticks for that repetition and std is returned as N/A. I attach the Excel file of BehaviorSpace’s output. Any help would be most appreciated!
I also encounter the same problem in the ‘environmental’ BehaviorSpace experiment, so I’m guessing the problem will be common to all the other pre-entered experiments too.
The stats output format aggregates data by tick, but the experiment you referenced outputs the final (and only) data point at a different tick for each run due to randomness in the model. Since it’s just a single row of values, I would recommend using Excel’s built-in functions such as MIN, MAX, and AVERAGE; BehaviorSpace’s stats output format is a bit overkill and not intended for this situation.
Thank you for your quick reply! That makes a lot of sense.
I created a quick fix by changing the stop condition to a standard value of 500 ticks, and having BehaviorSpace measure the number of ticks at the first instant when the population is fully infected. Stats output works fine with this version. Excel works too but this method has the benefit of getting BehaviorSpace to do the busy statistics work.
I attach the modified version containing the workaround in case it may be useful to anyone.
For real work, I have always used postprocessing in other tools (spreadsheets or Python programs), so I did not encounter this problem.
But I think it is perfectly reasonable to expect proper statistics outputs when recording results only at the end of run.
So, when “Run metrics every step” is off, the Stats Outputs should be as Derek expected.
The fact that it does not work intuitively for a model from the Models Library makes it even more important, I think.
Thank you for the feedback! It is certainly reasonable to expect proper statistics output when recording only the final tick, and that is the current behavior in most cases. The difference with this model is that each run ends at a different tick number, so it doesn’t make as much sense to aggregate those final values. Due to the fact that users can output metrics at arbitrary tick numbers that may be different between runs, and also because we allow the statistics to be read from either the spreadsheet or table output format, the only way to provide consistent and generally applicable statistics is to aggregate by tick number. If it turns out that this feature isn’t useful in the majority of cases, we can look into other ways to provide more useful summary statistics for BehaviorSpace experiments. Feel free to respond with any ideas you have for how we could provide more helpful statistics!
Thank you, Isaac. I understand that it might not be straight-forward to distinguish between the two cases.
Anyway, with “Run metrics every step” on there is no other proper way than aggregating by tick/step.
But, when “Run metrics every step” is off, there would be only one entry per run in both spreadsheet output and table output.
Would it be possible to ignore tick/step number in that case, and just aggregate over replications?
In fact, in one of the courses I was involved in, we use the Fire model to introduce BehaviorSpace (and sensitivity analysis in general). Then we are interested in the final number of burned trees, but also in how long it takes (the number of ticks). So, we take statistics over these outputs at the end of the run.
We did it already long before NetLogo 6.4, so our material did not use NetLogo’s Stats Outputs.
But, using Stats Outputs, I would expect something similar to the example above.
Can you look into a solution that differs for “Run metrics every step”?
Just to add on. I think there is a related issue with the Voting Sensitivity Analysis model. If we run the BehaviorSpace experiment with the following entered in the ‘vary variables as follows’ textbox:
[“initial-green-pct” [25 5 75]]
[“award-close-calls-to-loser?” false]
[“change-vote-if-tied?” false]
the stats output will again not work. In this case the stats output will not contain any numbers at all. Stats output is meant to generate Fig 7.6 of the book in this BSpace experiment, showing the mean final green percentage for initial green percentages ranging from 25 to 75 in steps of 5. The problem goes away if I just delete the two lower lines above, so that the ‘vary variables as follows’ box in BehaviorSpace just reads:
[“initial-green-pct” [25 5 75]].
In this case, the sliders on the Interface can be used to control the two Boolean variables that just got deleted. The problem seems to be that the textbox cannot accept Boolean variables for stats output?
This is not a big problem since the above workaround works well, but it would be nice if we could include the Boolean variables in the textbox and still use the Stats output, since this would allow us to conveniently record the Boolean variables in the file output of Stats output. Would this be possible?
