Code testing tools

Feature Requests
1

Our recent interviews with NetLogo users identified tools for testing code as a development need. And recent literature on good practices in agent-based modeling identifies the lack of code testing, or even a culture of testing, as a major concern–a sign that agent-based modeling is not yet a mature approach to science. Building tools for testing code into NetLogo would make it easier to test code, and would promote a code-testing culture by reminding users that code is not finished until it’s tested.

Here I describe an idea I have for a code testing tool, and I look forward to seeing other ideas.

I often write custom code to test a particular submodel by executing the submodel over wide ranges of values for several different inputs. For example, I might test a fish growth submodel using a test procedure that writes to file the growth rate calculated by the submodel over wide ranges of food intake, swimming speed, temperature, and fish size. Then I analyze that output in Excel (etc.) to search for errors.

I can envision a tool somewhat like BehaviorSpace that would produce this kind of test output. It would let users:

  1. Create local variables that identify specific turtles and/or patches, for example:
  • let a-fish one-of turtles
  • let fishs-patch [patch-here] of a-fish
  1. Create other local variables that are used as input to the code being tested, and specify ranges of values of those local variables (or observer variables). The BehaviorSpace syntax for specifying ranges of values could be used. For example:
  • let fish-lengths [5 5 25]
  • let patch-foods [0 1 20]
  • let temperatures [0 2 30]
  1. Provide a block of code that updates the submodel for each combination of input variable values:
  • ask a-fish [set length fish-lengths]
  • ask fishs-patch [set food patch-foods]
  • set temperature temperatures ; temperature is an observer variable
  • ask a-fish [update-food-intake] ; execute a procedure affecting growth
  1. Specify the outputs to be reported for each combination of input values, such as:
    [length] of a-fish
    [food] of fishs-patch
    temperature
    [growth] of a-fish ; A turtle reporter that calculates growth

Then of course the user could execute the tester so it creates an output file with all the results. The output file could also produce metadata documenting exactly what code was executed how.

1 Like
2

Great idea! I wonder if we could start by building an extension/library for this?

3

I agree this is a great idea. It seems what you are proposing here is an easier way to do meso- and macro-level testing as per the framework in this paper: A generic testing framework for agent-based simulation models | Journal of Simulation

I totally agree and also think we need better tools and best-practices for unit-testing as well (micro-level testing).

There is also work on “property-based testing” that we should look into and potentially build tools for: Specification testing of agent-based simulation using property-based testing | Autonomous Agents and Multi-Agent Systems

Regarding John’s point, I think all of these things could be done as extensions.

4

I have now drafted two concepts for code testers. One would be BehaviorSpace-like and test a piece of code over wide ranges of selected variables that affect it. The second would be an extension (perhaps) that records the values of selected variables at the start and end of a selected procedure. The NetLogo development team has these concept descriptions.

1 Like
5

What do you mean by the “concept descriptions” we have?

1 Like
6

For the record, there have been previous attempts at doing this:

7

The absence of a testing tool makes it challenging not only to validate NetLogo code but also to publish it. As an illustration, here’s an example from an open review:

One workaround is to use the R package logolink to programmatically run NetLogo experiments from R, combined with testthat to define expectations and check results. While this isn’t a built-in testing suite for NetLogo, it can serve a similar purpose.

3 Likes
8

This is good to know about. We’ll keep it in mind for the future as we think more about potential tools for testing. Is this something you would be interested in contributing to?

9

Yes. I’ll start by working on this R pipeline before proposing a new test extension.

1 Like
10

Note ( 2025-12-03):

netlogo-check is now part of LogoActions For details, see the section Running Experiments with Quarto and logolink.

I put together a prototype workaround for running NetLogo simulations through GitHub Actions. You can check the repo here:

https://github.com/danielvartan/netlogo-check

It also publishes the test results to GitHub Pages:

https://danielvartan.github.io/netlogo-check/tests/qmd/spread-of-disease.html#population-density-runtime

Instead of using nlrx, I went with logolink. It works fine. You only need to add testthat to define the expectations.

My plan is to release a dedicated GitHub Action that installs and configures NetLogo on Ubuntu runners. For now, the setup step is simple:

- name: Set up NetLogo
  run: |
    wget https://ccl.northwestern.edu/netlogo/7.0.2/NetLogo-7.0.2-64.tgz
    sudo tar -xvzf NetLogo-7.0.2-64.tgz -C "/opt/"

By installing NetLogo in /opt/, logolink will automatically detects the executables, so there’s nothing else to configure.

2 Likes
11

I recently launched LogoActions , which makes creating unit tests in NetLogo much easier:

It’s not a testing suite, but I believe it’s a significant step toward more sophisticated tools.

1 Like
12

Nice! Are you using this to run unit tests or more to see if data output is what you expect? (Maybe you don’t consider those to be different)

Also, your links on the second to last post seem to be broken…

13

Hi Jacob!,

There are several ways to use LogoActions. I’m using it for all the cases below.

One is the one you mentioned: running and integrating automated tests for BehaviorSpace experiments.

Another is generating reproducible reports that depend on NetLogo simulations. That is where Quarto and logolink come in.

You can also create unit tests by defining multiple experiments inside those reports, but I do not think that is the ideal setup. NetLogo still needs a proper test framework like testthat, although this moves things a bit in that direction.

A bigger use case is automating large experiment runs on large runners. If someone has a GitHub Pro account (free for students and academics), they can already run heavier experiments directly in their repositories and just collect the results.

About the broken links: I merged netlogo-check intoLogoActions, so the old links no longer work. I added a note in the post informing this. The netlogo-check workflow is shown in the section Running Experiments with Quarto and logolink.

Cheers!

14

Awesome! I didn’t know about GitHub Pro and being able to run heavier experiments there. That’s super useful to know and something we’ll look into.

I agree that NetLogo still needs a proper test framework.

1 Like
15

I just released the check-netlogo action in the LogoActions project, which solves most of the issues we’ve been discussing.

Together with setup-netlogo, it automatically checks whether all BehaviorSpace experiments in a repository’s NetLogo models run without errors. This lets you run parameter sweeps on every commit to the main branch and catch problems introduced by changes to models or experiments.

Below is a basic workflow configuration. To use it, create a file named check-netlogo.yaml with the content below and place it in .github/workflows at the root of your repository.

@jzkelter, you can use it in GitHub - NetLogo/models: NetLogo Models Library. It will check all model experiments in the Model Library on every commit.

on:
  push:
    branches: [main, master]

name: "NetLogo Check"

permissions: read-all

jobs:
  check-netlogo:
    runs-on: ubuntu-latest
    name: Check NetLogo Models
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up NetLogo
        uses: danielvartan/logoactions/setup-netlogo@v1

      - name: Check NetLogo models
        uses: danielvartan/logoactions/check-netlogo@v1

It’s still not a full test suite, like testthat, but it gets close.

I’m using this workflow with LogoClim and it runs fast. One of the runs is shown here:

It ran 338 simulations with a total of 5859 steps, each step loading a ~700 kB raster. The run finished in 6 minutes and 51 seconds.

The action also produces an artifact containing the output tables for each experiment.

For parameter testing, this workflow already does the job. What it doesn’t handle:

  • Error behavior testing (e.g., argument validation, like in checkmate)
  • Function (report/procedure) behavior testing
  • Error tolerance testing (e.g., for models that import data; example)

These can be handled for now with Quarto and logolink (see this workflow), though it’s not ideal. A proper test package is still needed.

1 Like
16

So glad to see this! Testing is one of those things that usually falls by the wayside in ABM, but these tools make it so much more approachable. The GitHub Actions integration is exactly what the community needs to build a proper 'code-testing culture.

2 Likes
17

For those that may be struggling with this subject, I suggest checking the solutions I presented to JOSS reviewers to verify the functionality of the LogoClim model: