  Continuous integration (CI) is a widely used practice in modern software
engineering. Unfortunately, it is also an expensive practice - Google and
Mozilla estimate their CI systems in millions of dollars. There are a number of
techniques and tools designed to or having the potential to save the cost of CI
or expand its benefit - reducing time to feedback. However, their benefits in
some dimensions may also result in drawbacks in others. They may also be
beneficial in other scenarios where they are not designed to help. In this
paper, we perform the first exhaustive comparison of techniques to improve CI,
evaluating 14 variants of 10 techniques using selection and prioritization
strategies on build and test granularity. We evaluate their strengths and
weaknesses with 10 different cost and time-tofeedback saving metrics on 100
real-world projects. We analyze the results of all techniques to understand the
design decisions that helped different dimensions of benefit. We also
synthesized those results to lay out a series of recommendations for the
development of future research techniques to advance this area.
