  We propose a novel approach to the statistical analysis of simulation models
and, especially, agent-based models (ABMs). Our main goal is to provide a fully
automated and model-independent tool-kit to inspect simulations and perform
counterfactual analysis. Our approach: (i) is easy-to-use by the modeller, (ii)
improves reproducibility of results, (iii) optimizes running time given the
modeller's machine, (iv) automatically chooses the number of required
simulations and simulation steps to reach user-specified statistical
confidence, and (v) automatically performs a variety of statistical tests. In
particular, our framework is designed to distinguish the transient dynamics of
the model from its steady-state behaviour (if any), estimate properties of the
model in both "phases", and provide indications on the ergodic (or non-ergodic)
nature of the simulated processes -- which, in turns allows one to gauge the
reliability of a steady-state analysis. Estimates are equipped with statistical
guarantees, allowing for robust comparisons across computational experiments.
To demonstrate the effectiveness of our approach, we apply it to two models
from the literature: a large scale macro-financial ABM and a small scale
prediction market model. Compared to prior analyses of these models, we obtain
new insights and we are able to identify and fix some erroneous conclusions.
