Abstract This paper presents a new approach to test the performance of protocols, services, or application implementations (IUT) under normal and overload situations, to identify performance levels of the IUT for ranges of parameter settings and to assess the measured performance. A performance test describes precisely the configuration of the IUT, the configuration of the network, and the load characteristics (the so called performance test configuration). PerfTTCN -an extension of TTCN with notions of time, measurements, and performance -is a formalism to describe performance tests in an understandable, unambiguous and re-usable way with the benefit to make the test results comparable. Different approaches for guaranteeing the end user certain levels of QoS were developed, since the non-functional aspects, in particular Quality-of-Service (QoS) aspects of distributed telecommunication services (e.g. multimedia collaboration, tele-teaching, etc.) are as important as functional aspects. They include approaches for QoS negotiation between peer user and service and network provider, QoS guarantees of communication services, QoS monitoring and management including self-adapting applications. This proposal considers QoS in the area of testing. In general, testing is a method to check whether an implementation meets certain requirements that are described in a specification. QoS testing checks the end-to-end quality of a service implementation against the QoS requirements. A specific class of QoS is that of performance-oriented QoS including delays (e.g. of a response), throughputs (e.g. for bulk data), and rates (e.g. of errors). We concentrate exclusively on performance-oriented QoS, other classes of QoS are out of the scope. Subsequently, we use the term performance instead of QoS and consequently performance testing instead of QoS testing. One of the well-established methods in testing is that of conformance testing, which is used to check that an implementation meets its functional requirements. Since conformance testing is aimed at checking purely the functional behaviour of system implementations, it lacks in concepts of time and performance. Timers are the only way to require certain time periods for test events to occur. They are used to determine whether test events occur too early, too late or not at all. While traditionally the temporal ordering and type of PDUs/ASPs have been the main target of conformance testing, we attempt here to introduce performance measurements and Quality of Service (QoS) requirements in the conformance testing. Performance measurements in a 1. Guest researcher from the University of Ottawa, Canada 2 network traditionally consists in sending time stamped packets through a network and record delays and throughput. Once such data has been collected a number of statistics can be computed and displayed. However these statistics can sometime be meaningless when the actual conditions in which these measurements have been performed are unknown. Different strategies can be used to study operational conditions of a network. One consists in attempting to analyse real network traffic load and correlate it with the test results and the other method consists in creating artificially specific network traffic load and correlate it directly to specific behaviours observed in the performance test. The first method enables to study performance under real traffic conditions and confront unexpected behaviours while the second method enables more precise scientific measurements because the conditions of an experiment are fully known and controllable and correlations with observed performance are less fuzzy as with real traffic. Both methods are actually useful and somewhat complementary and a testing cycle involving both methods can be considered by exploring new behaviours with real traffic load and then attempting to reproduce them artificially to further refine their understanding with the help of the second method. The presented approach to performance testing attempts to address both methods. Although this method is quite general, one of its primary goals is for the study of ATM network performance at the application layer. The approach used in this work is in line with the ATM Forum performance benchmarking specifications [ATM96] that are also interested to measure performance not at the ATM cell level but by frame-level performance and performance perceived at higher layers. The goals of performance testing can be achieved with a variety of existing languages and tools. TTCN [CMTF, TTCN] is the only standardized, well known and widely used notation for the description of conformance tests, for which a number of tools are available. We decided to base our work on TTCN due to its wide acceptance. We define an extension of the TTCN language to handle performance testing and explore the integration of traffic load models into the TTCN language. In both cases there are limited additional declaration constructs necessary to achieve this goal. Our proposal introduces a new consideration of time within the TTCN language. So far, the current standard considers time exclusively in timers where the execution of a test can be branched out to an alternative path if a given timer expires. New proposals by [KIV97] introduce concepts of local and global time constraints during the execution of a dynamic behaviour. Our proposal consists in gathering substantial samples of time measurements between selected test events and compute various statistics on determined sample sizes to be used in the evaluation of performance characteristics. These characteristics are then used to verify their conformance to performance constraints based on QoS criteria. In contrast to current TTCN, a performance constraint evaluation is based on repeated occurrences of pairs of events rather than the evaluation of a single event. In this paper, we first introduce the objectives, main concepts, and architecture of performance testing, next we present the language features to describe the new concepts in TTCN, and finally we present some results of experiments on an example handling queries to an HTTP server using a modified test generator of some well known TTCN design tool.  Introduction to performance testing The primary goal of performance testing is to test the performance of an implementation under normal and overload situations. The normal and overload situations are described with a performance test configuration, which identifies the configuration of the IUT, the configuration of the network, and the load characteristics. A notation for the description of performance tests is needed, that should have a well-defined syntax and an operational semantics in order to reduce the possibilities of misinterpretations. That notation for performance tests is a precondition to make performance tests understandable, unambiguous and re-usable. The main advantage of our method is to make the test results comparable.This is in contrast with informal methods where test measurement results are provided only with a vague description of the measurement configuration, so that it is difficult to re-demonstrate and to compare the results precisely. Once more it has to be noted that the mere fact that these tests are described formally ensures unambiguity and re-usability. Another goal of performance testing is to identify performance levels of the IUT for ranges of parameter settings. Several performance tests will be executed with different parameter settings. The testing results are then interpolated in order to adjust that range of parameter value where the IUT shows a certain performance level. For performance testing, the conformance of the IUT is assumed. However, since overload may degrade the functional behaviour of the IUT to be faulty, care has to be taken to capture erroneous functional behaviour in the process of performance testing as it is done in conformance testing. Finally, if performance requirements for the IUT are given, performance testing should result in an assessment of the measured performance, i.e. does the implementation under test meet the performance requirements or not. This section discusses the basic concepts of the performance test approach. The concepts are separated w.r.t. the test configuration, measurements and analysis, and test behaviour. A performance test consists of several, possibly distributed foreground and background test components. They are coordinated by a main tester (control component). A foreground test component realizes the communication with the IUT. It directly influences the IUT. The foreground tester uses discrete test events, which are the same test events that are used in conformance testing, in order to drive the IUT into specific states in which the performance measurements are executed. Once the IUT is in a state that is under consideration for performance testing, the foreground tester issues a continuous stream of data packets to emulate the foreground load for the IUT. The foreground load is also called foreground traffic. It is described by means of traffic models. 4 A background test component generates continuous streams of data in order to emulate the load of the IUT. A background tester does not directly communicate with the IUT. It only implicitly influences the IUT as it brings the IUT into minimal, normal, or overload situations. The background traffic is also described by means of traffic models. Traffic models describe the communication data as a stochastic stream of data packets with varying interarrival times and varying packet length. An often used model for the description of traffic is that of Markov Modulated Poison Processes (MMPP[ONV94] ). We selected this model for traffic description due to its generosity (e.g. a number of different multimedia streams such as audio and video have been described as MMPPs) and efficiency (e.g. efficient random number generator and an efficient finite state machine logic are needed only). Nonetheless, the performance testing approach is open to other kinds of traffic models. Points of Control and Observation (PCOs) are the access points for the foreground and background test components to the interface of the IUT. They offer means to exchange SDUs or PDUs with the IUT and to monitor the occurrence of test events (i.e. to collect the time stamps of test events). A specific use of PCOs are their use for monitoring purposes only. Monitoring only is needed to observe for example the artificial load of the background test components, the load of real network components that are not controlled by the performance test or to observe the test events of the foreground test component. To sum up, a performance tests uses an ensemble of foreground and background tester with the used traffic models and points of control and observation, and points of measurements. The performance test configuration describes unambiguously the conditions under which the measurements are executed. It is the description of the performance test configuration that makes performance test experiments re-usable and performance test measurements comparable. The test components of a performance test may be distributed in the communication network and may use load generators to generate background traffic and monitors to collect time stamps and to execute measurements. A main tester is used to coordinate the test activities of the tester. One can identify different classes of performance test configurations in dependence of the type of the implementation under test. For example in the case of performance testing in communication networks, we distinguish between performance testing the implementation (either in hardware, software, or both) of • an application, • an end-to-end transmission service, and • a protocol. The test configurations for these three types of performance tests are given in Figure 1 , 2, and 3. The notion System Under Test (SUT) comprises all IUT and network components. For simplification, we omit the inclusion of a main tester in the figures.The test configurations differ only in the use of foreground tester. The usage of background tester to generate artificial load to the network, i.e. to bring the network into normal load or overload situations, and measurements on the actual real load in the network, i.e. to monitor the load of real network 5 applications (as opposed to the artificial background tester), is the same in each of the test configurations. In the case of performance testing of a server (Fig. 1) , foreground tester emulate the clients. That test configuration for an end-to-end service (Fig. 2) includes foreground tester at both ends of the end-to-end service, which emulate the service user. Performance testing of a protocol (Fig. 3) with a distributed test method (please refer to [CTMF] for other test methods) includes foreground tester at the upper service access point to the protocol under test and at the lower service access point. The service access points are reflected by points of control and observation. A measurement is based on the collection of time stamps of events. One has to describe precisely the format of each event that belongs to a measurement, so that the time stamp can be collected whenever an event at a certain PCO matches that format. A measurement is started once and continues until it is explicitly cancelled or reaches the time duration. Based on the measurements, more elaborated performance characteristics such as mean, standard deviation, maximum and minimum as well as the distribution functions can be evaluated. These are supported by so called metrics. The evaluation of performance characteristics is supported either by an on-line analysis (i.e. during test execution) or in an off-line analysis (i.e. after the test finishes and all samples have been collected). On-line analysis is in particular needed for performance constraints that allow one to describe requirements on the observed performance so that either a pass or a fail verdict can be assigned. A foreground test components has to offer features on starting and cancelling background test components, on starting and cancelling measurements, on generating controlled load to the IUT, and on accessing the recent measurements via performance constraints. Finally, performance verdicts should be assigned so that they do not only assess the observed behaviour and performance of the IUT to be true or false, but also that give back the measured performance characteristics being of importance for the test report.  PerfTTCN -a performance extension of TTCN Since background test traffic depends on a particular test configuration its localization and orientation is to be added to a test component configuration declaration (Table 1) . For each background test traffic instance PCOs should be defined to identify the location for the test generation (left side) and its destination (right side). In addition, PCO lists are foreseen to integrate multipoint transmissions. The related coordination points are necessary for the control of the traffic generator and any traffic monitor. These components are controlled by coordination messages within the dynamic test behaviour. The qualities of the background traffic are defined by traffic models which are referred in Background Traffic Stream Declarations (see next section). Specific details on e.g. connection information such as VPI/VCI for an ATM connection are subject to the PCO PIXIT document information. The purpose of the background traffic is to create load on a network that will be traversed by the communication links of the system under test. The background traffic is a constant uninterrupted and predictable stream of packets following a given stochastic pattern of transmission times. One of the main characteristics of the background traffic is to follow specific rate patterns that follow stochastic laws. The purpose of these patterns is to simulate the traffic associated with different kinds of applications. The two essential components of a traffic load are the dataflow configuration and the stochastic models declarations. These two components are then combined in the background traffic stream declarations using as many instances of dataflow/stochastic models combinations as necessary to produce significant load. This consists in selecting the number of instances of each dataflow/model type combination. Each of these traffic streams is identified by a name that can be used in the dynamic behaviour part to start the appropriate traffic load. The traffic model declaration is used to select stochastic models and set their corresponding
-PerfTTCN (Performance TTCN ) REF extends TTCN with concepts for performance testing, such as: performance test scenarios for the description of test configurations, traffic models for the description of discrete and continuous streams of data, measurement points as special observation points, measurement declarations for the definition of metrics to be observed at measurement points, performance constraints to describe the performance conditions that should be met, performance verdicts for the judgement of test results.
PerfTTCN, a TTCN language extension for performance testing
Testing of Communicating Systems
Testing of Communicating Systems
Computer Science