Prioritizing versions for performance regression testing: The Pharo case Abstract   Context  Software performance may suffer regressions caused by source code changes. Measuring performance at each new software version is useful for early detection of performance regressions. However, systematically running benchmarks is often impractical (e.g., long running execution, prioritizing functional correctness over nonfunctional).    Objective  In this article, we propose Horizontal Profiling, a sampling technique to predict when a new revision may cause a regression by analyzing the source code and using runtime information of a previous version. The goal of Horizontal Profiling is to reduce the performance testing overhead by benchmarking just software versions that contain costly source code changes.    Method  We present an evaluation in which we apply Horizontal Profiling to identify performance regressions of 17 software projects written in the Pharo programming language, totaling 1,288 software versions.    Results  Horizontal Profiling detects more than 80% of the regressions by benchmarking less than 20% of the versions. In addition, our experiments show that Horizontal Profiling has better precision and executes the benchmarks in less versions that the state of the art tools, under our benchmarks.    Conclusions  We conclude that by adequately characterizing the runtime information of a previous version, it is possible to determine if a new version is likely to introduce a performance regression or not. As a consequence, a significant fraction of the performance regressions are identified by benchmarking only a small fraction of the software versions.

Accurate Modeling of Performance Histories for Evolving Software Systems Learning from the history of a software systemu0027s performance behavior does not only help discovering and locating performance bugs, but also identifying evolutionary performance patterns and general trends, such as when technical debt accumulates. Exhaustive regression testing is usually impractical, because rigorous performance benchmarking requires executing a realistic workload per revision, which results in large execution times. In this paper, we propose a novel active revision sampling approach, which aims at tracking and understanding a systemu0027s performance history by approximating the performance behavior of a software system across all of its revisions. In a nutshell, we iteratively sample and measure the performance of specific revisions that help us building an exact performanceevolution model, and we use Gaussian Process models to assess in which revision ranges our model is most uncertain with the goal to sample further revisions for measurement. We have conducted an empirical analysis of the evolutionary performance behavior modeled as a time series of the histories of six realworld software systems. Our evaluation demonstrates that Gaussian Process models are able to accurately estimate the performanceevolution history of realworld software systems with only few measurements and to reveal interesting behaviors and trends.