Please excuse me for bringing up yet another BehaviorSpace issue.
This is something I noticed a long time ago, but I had forgotten about. Now, I encountered it again.
When the model code causes a NetLogo runtime error, BehaviorSpace pops up an error but continues the run. It looks like the current step is aborted and the next step (Go commands) starts immediately.
I don’t know whether this is intended behavior or it is actually a bug.
But, I think it does not make much sense continuing the current run when a runtime error has occurred.
It could even result is huge run times without valid results.
So, would it be possible to abort the current run when a runtime error occurs?
Continuing the job at the next run could still be useful.
No need to apologize, it’s always useful to get specific and thoughtful feedback about a feature! I do believe that it’s the intended behavior for the run to continue, but I agree that it doesn’t always make sense. I’m not sure if it’s the right move to completely change that behavior, but it would be very reasonable to have an “Abort Run” button (and maybe also an “Abort Experiment” button) in the runtime error dialog. Would this be a satisfactory solution?
If this was indeed intended behavior, it would be akward changing it. Although, I don’t see why continuing a broken run would be useful.
For your suggestion to work at all, the dialog should also block the current run, which it does not do currently.
Another reason why I would not be happy with this solution is that the job won’t continue when one run has an error. Often, for long jobs, I would start them before lunch to find the results later. Then, it would be annoying to find that run 20 (out of 1000) blocked the whole job.
[In fact, I have been starting jobs at the office before going home, to find the results the next day.]
Would an option in the dialog for starting a job be a solution?
Something like a checkbox “Abort runs with runtime errors” and/or a checkbox “Abort whole experiment when a runtime error occurs”.
Good point about the blocking issue. A checkbox in the run options dialog sounds good to me; if the rest of the devel team agrees then we’ll add it to a future release!
It’s not obvious to me why we would want to run allow the run to continue if it errored. Presumably, you’re running BehaviorSpace to gather data from your model. If an unhandled (and, thus, probably unexpected) error is encountered… it’s hard to imagine that your data is meaningful. By continuing with the run, it seems to me like we’re just driving up the user’s energy bill, at that point, to no one’s benefit.
I’m inclined to think that errors should flat-out abort the experiment.
I can be easily convinced that the default behavior should be for the erroring run to be aborted, but I don’t think we should abort the entire experiment. Especially with longer experiments that have unrelated runs, I might just want to let it go and get all the data I can, filtering out outliers and bad data after the fact. If the entire experiment gets aborted just because one run had some sort of edge case bug, I would miss out on all the good data from other runs.
Thank you for your insights. Indeed, I think an unhandled error renders the rest of the run useless.
But that does not necessarily hold for the whole experiment. Good point about the energy bill, though.
There are many cases, where other runs from the experiment can be valuable. One is when the error occurs for unexpected combinations of parameter values, while the model has been tested for other parameter combinations. Another is to see for which settings the (unexpected) error occurs as well.
Therefore, I would go for aborting the current run, but not automatically the whole experiment.
Maybe, a checkbox could control if the whole experiment should be aborted as well?
After discussing the issue, we have decided to add a dropdown menu to the BehaviorSpace run options dialog, allowing you to choose whether to ignore runtime errors, abort the run, or abort the experiment. This will most likely be implemented in NetLogo 7.1. Thank you again for the suggestions!
Yes, it will be implemented for headless runs as well. There will be a command line flag that will provide equivalent functionality, for example something like --error ignore or --error abort-run.