How can simulation programs be validated?
Microscopic agent-based simulation software is based on scientific models influenced by physics, mathematics, psychology, traffic engineering, and more. Simulations are generally used to study complex relationships. They reduce reality to the relevant aspects in order to study effects. Various validation methods are used to ensure that the selected model is sufficiently realistic.
There are several ways to validate microscopic agent-based simulators
Simulation programs should include scientifically validated metrics. In crowd:it, for example, average values for walking speed or body circumference are included that have been collected in studies such as the one conducted by Weidmann. A detailed explanation of the fundamental diagram can be found here.
Especially for the specific motion behavior of humans, results from real experiments, e.g. from the Jülich Research Center for the research project BaSiGo, should be included in the software. In general, a distinction is made between real experiments and controlled experiments.
Test suites consist of a selection of simple test cases that can be run against the software. They provide information about whether the software can actually meet the necessary requirements.
- The RiMEA test suite; the most important in Germany (see the next section).
- ISO-20414: An international norm that defines several test cases for microscopic simulation tools.
- In the US, NIST tests (NIST technical note 1822) are used used.
- For the maritime sector, ISO 1238 exists
Currently, work is underway to unify these tests under the ISO standard in Chapter Fire Safety Engineering.
Validation of crowd:it
RiMEA tests on a regular basis
Since its foundation in 2014, accu:rate has been a member of RiMEA e.V. (Guideline for Microscopic Evacuation Analysis). The goal of RiMEA is to support authorities and those developing microscopic pedestrian simulators with a guideline. It defines minimum standards for input parameters and simulation models.
Appendix 1 of the guideline presents a number of test cases. These test cases allow a user of simulation software to better understand the results of their simulation.
We run these test cases with crowd:it on a regular basis, enabling you to better understand our simulation model. Transparency is one of our key values! On request, we will also be happy to provide you with our older test cases. Simply contact us for this.
Comparison of simulation results with experiments
We regularly compare the results of a crowd:it simulation with results from both real and controlled experiments. Here we observed an evacuation exercise, before simulated the same scenario with crowd:it. There was less than 5% deviation in evacuation time between crowd:it and the experiment. In order to further develop our staircase model, we carried out both real and controlled experiments in collaboration with the University of Applied Sciences Munich, whose results we subsequently used for validation.
Comparison of simulation results with other calculation methods
In order to compare the results of crowd:it against macroscopic simulation methods, we considered th capacity analyses and hydraulic methods of Predteschenski & Milinski.
The evacuation times of a three-storey office building with a 1.20 m wide staircase and simple floor plan is shown in the following diagram:
crowd:it lies between the respective procedures and unlike macroscopic procedures, crowd:it achieves exact results with low agent numbers.