Computational models of complex systems are usually elaborate and sensitive to implementation details, characteristics which often affect model verification and validation. Model replication is a possible solution to this problem, as it bypasses the biases associated with the language or toolkit used to develop the original model, promoting model verification , model validation, and improved modelunderstanding. Some argue that a computational model is untrustworthy until it has been successfully replicated. However, most models have only been implemented by the original developer, and thus, have never been replicated. Several reasons for this problem have been identified, namely: a) lack of incentive; b) below par modelcommunication; c) insufficient knowledge of how to replicate; and, d) level of difficulty of the replication task. In this paper, we present a model comparison technique, which uses principal component analysis to convert simulation output into a set of linearly uncorrelated statistical measures, analyzable in a consistent, model-independent fashion. It is appropriate for ascertaining distributional equivalence of a model replication with its original implementation. Besides model-independence, this technique has three other desirable properties: a) it automatically selects output features that best explain implementation differences; b) it does not depend on the distributional properties of simulation output; and, c) it simplifies the modelers' work, as it can be used directly on simulation outputs. The proposed technique is shown to produce similar results to classic comparison methods when applied to a well-studied reference model.
Computational models of complex systems are usually elaborate and sensitive to implementation details, characteristics which often affect their verification and validation. Model replication is a possible solution to this issue. It avoids biases associated with the language or toolkit used to develop the original model, not only promoting its verification and validation, but also fostering the credibility of the underlying conceptual model. However, different model implementations must be compared to assess their equivalence. The problem is, given two or more implementations of a stochastic model, how to prove that they display similar behavior? In this paper, we present a model comparison technique, which uses principal component analysis to convert simulation output into a set of linearly uncorrelated statistical measures, analyzable in a consistent, model-independent fashion. It is appropriate for ascertaining distributional equivalence of a model replication with its original implementation. Besides model-independence, this technique has three other desirable properties: a) it automatically selects output features that best explain implementation differences; b) it does not depend on the distributional properties of simulation output; and, c) it simplifies the modelers' work, as it can be used directly on simulation outputs. The proposed technique is shown to produce similar results to classic comparison methods when applied to a well-studied reference model.
R
model
their
R
problem, as it bypasses the
issue. It avoids
R
promoting model verification , model validation, and improved modelunderstanding. Some argue that a computational model is untrustworthy until it has been successfully replicated. However, most models have only been implemented by the original developer, and thus, have never been replicated. Several reasons for this problem have been identified, namely: a) lack of incentive; b) below par modelcommunication; c) insufficient knowledge of how to replicate; and, d) level of difficulty of the replication task.
not only promoting its verification and validation, but also fostering the credibility of the underlying conceptual model. However, different model implementations must be compared to assess their equivalence. The problem is, given two or more implementations of a stochastic model, how to prove that they display similar behavior?