An important design decision in the construction of a simulator is how to enable users to access the data generated in each run of a simulation experiment. As the simulator executes, the samples of performance metrics that are generated beg to be exposed either in their raw state or after having undergone mathematical processing. Also of concern is the particular format this data assumes when externalized to mass storage, since it determines the ease of processing by other applications or interpretation by the user. In this paper, we present a framework for the ns-3 network simulator for capturing data from inside an experiment, subjecting it to mathematical transformations, and ultimately marshaling it into various output formats. The application of this functionality is illustrated and analyzed via a study of common use cases. Although the implementation of our approach is specific to ns-3, this design presents lessons transferrable to other platforms. In computer simulation, data is both cause and consequence. The abstractions of a system or of a process that are used in simulation, known as models, are derived from the analysis of significant body of data. This "raw material" is most often acquired through heavy, challenging work and serves as the input that drives modeling activities. Once models are defined and subjected to an iterative process of validation and verification, their computational implementations are used in experiments executed by a simulator, which produces output data. This second body of data is processed and analyzed to yield answers for questions posed at the start of the simulation study. The collection and the processing of input and output data present significant challenges, each in its own way. In this paper, we focus exclusively on problems related to the generation and, to some extent, on the processing of output data. Many simulators are constructed specifically for one particular simulation study and their construction tends to pay little regard to software engineering practices that promote flexibility and extensibility. The mechanisms for generation of output data, in these simulators, can be as simple as the direct use of a programming language's I/O functionality to generate streams of samples of a few metrics of interest in the model. The instrumentation of code to externalize these samples may be done on a case by case basis, without the application of design patterns of any kind. Moreover, much of the processing of these samples Perrone, Henderson, Watrous, and Felizardo may be done internally to the simulator and often with code written "from scratch." It comes as no surprise that many studies derived from simulators like these lack credibility, as they are prime examples of the situations identified as pitfalls by Uhrmacher (2012). The situation is quite different in the case of simulators whose lifespan outlasts that of a single study, where one may find sophisticated, reliable functionality for data collection and processing. Still, as indicated by Ribault et al. (2010) , many simulators provide APIs for data collection that require the mixing of instrumentation code into model code. There are two dangers in this approach. On the one hand, the model programmer may "overinstrument" the code to provide proactively for all possible and imaginable observation needs. This practice could lead to longer simulation runs that generate much more data than what is required for the average study. On the other hand, "underinstrumenting" models could require their future users to dive in and modify code, what could possibly introduce errors in the model and compromise experiments. In any case, the fact is that the quality of a simulation study is determined not only by how adequate the granularity and the amount of data collected are, but also by how easy it is to reproduce it. Kurkowski, Camp, and Colagrosso (2005) call attention to the importance of bundling data collection and output analysis into the simulation run, which can benefit reproducibility. Recent literature, such as (Helms et al. 2012) , indicate a growing attention to the search for simulator designs that include data collection mechanisms which are easy to use, flexible, and efficient, both in terms of memory and run time, and which can be coupled with some type of output analysis. In this paper, we present a framework for the ns-3 network simulator that works toward these requirements. Although our framework relies on specific ns-3 functionality, we expect that its underlying design provides lessons that can be generalized for application in other simulation platforms. The remainder of the paper is structured as follows. Section 2 establishes a context for our contribution by presenting a few notable examples of related work. Section 3 expands this background by exposing the reader to the fundamentals of the design of ns-3 and to the motivation behind our work. Section 4 introduces design goals that drove the design of this data collection framework and also the foundation classes that define it. Section 5 discusses implementation issues and shows examples of the application of the framework. Finally, Section 6 outlines ongoing work and future directions to conclude the paper.
SAFE also provides automated data processing, data storage, and data visualization functionalities that are complemented by the ns-3 data collection framework (DCF) REF .
The design of an output data collection framework for NS-3
2013 Winter Simulations Conference (WSC)
2013 Winter Simulations Conference (WSC)
Computer Science