A well-used aphorism from British Mathematician Dr. George Box is, "All models are wrong. Some are useful." (George E. P. Box) The reason they are wrong is that models are a simplified representation of some real system, often a complex one. Repeatability is therefore an important component in discovering a model's validity. Yet there is some evidence that there is a crisis in the world of science in that many results cannot be reproduced.
Drummond pointed out that reproduction and replication are not the same thing. (Replicability is not Reproducibility: Nor is it Good Science) He referred to replicability as the "poor cousin" of reproducibility, pointing out that reproducibility requires some changes to the experiment's environment while replication studiously avoids these changes. Drummond was making his point from the context of machine learning where two model runs will often create slightly different results. This is also the case with agent-based models and one of the reasons that these models are run multiple times before the collective results are analyzed. Advocates of replication also insist that sharing computer program code will allow for precise repeatable results and that such precision is the desired goal. Reproducibility, however, permits variations in the results that can point to flaws in the original model that lend to its falsifiability as well as to incremental improvements to the original model's results. Reproducibility does not require sharing of computer code, merely the sharing of the process description.
In 2007 Wilensky and Rand (Making Models Match: Replicating an Agent-Based Model) suggested a method to recreate a model and account for the variations in the reproduction process. (Here they do use the word replicate when reproduce would be a better choice.) They noted six areas in which a new model can incur sufficiently significant changes leading to an independent reproduction of results compared to the original:
- Time: The reproduction occurs at a later date. This is the change least likely to create different results.
- Hardware: The physical computing environment has been changed. At a minimum, a different machine is used. More likely, the model is reproduced on a more capable computer.
- Languages: A high-level computer language such as FORTRAN, LISP, C++, or NetLogo is used that is not the same as the original.
- Toolkits: A set of libraries, specific to the computational tasks at hand, are changed even if the computer language is not.
- Algorithms: An inner process is changed such as the agent activation scheme, the type of sorting algorithm, or the order in which the inner processes occur.
- Authors: The original authors are not involved in the reproduction.
Given a well-documented description of the original model's initial conditions, interactions between agents and their environment, and results obtained, it should be possible to arrive at similar conclusions with a newly created model.
In a future posting, I will discuss reproducing a model from the 1980's. The original authors made it rather easy even without access to the model's code.