This paper describes pitfalls, issues, and methodology for measuring software performance. Ideally, measurement should be performed and reported in such a way that others will be able to reproduce the results in order to confirm their validity. We aim to motivate scientists to apply the necessary rigor to the design and execution of their software performance measurements to achieve reliable results. Repeatability of experiments, comparability of reported results, and verifiability of claims that are based on such results can be achieved only when measurements and reporting procedures can be trusted. In short, this paper urges the reader to measure the right performance and to measure the performance right.
Pieterse et al. REF argue that researchers should ensure a rigorous design of their experiments and apply software performance measurements techniques to guarantee reliable results, so as to support the repeatability of their experiments, comparability of their reported results, and verifiability of their claims.
The ghost in the machine : don’t let it haunt your software performance measurements
Computer Science