Repository: rust
Issue: Tracking Issue for making incremental compilation the default for Release Builds
Labels: I-compiletime,T-compiler,A-incr-comp,T-core,T-cargo,C-tracking-issue,WG-compiler-performance,S-tracking-design-concerns

Since incremental compilation supports being used in conjunction with ThinLTO the runtime performance of incrementally built artifacts is (presumably) roughly on par with non-incrementally built code. At the same time, building things incrementally often is significantly faster (([1.4-5x](https://github.com/rust-lang/rust/pull/56678#issuecomment-446606215) according to perf.rlo). As a consequence it might be a good idea to make Cargo default to incremental compilation for release builds.

Possible caveats that need to be resolved:

- [ ] The initial build is slightly slower with incremental compilation, usually around 10%. We need to decide if this is a worthwhile tradeoff. For `debug` and `check` builds everybody seems to be fine with this already. 
- [ ] Some crates, like `style-servo`, are always slower to compile with incr. comp., even if there is just a small change. In the case of `style-servo` that is 62 seconds versus 64-69 seconds on perf.rlo. It is unlikely that this would improve before we make incr. comp. the default. We need to decide if this is a justifiable price to pay for improvements in other projects.
- [ ] Even if incremental compilation becomes the default, one can still always opt out of it via the `CARGO_INCREMENTAL` flag or a local Cargo config. However, this might not be common knowledge, the same as it isn't common knowledge that one can improve runtime performance by forcing the compiler to use just one codegen unit.
- [x] It still needs to be verified that runtime performance of compiled artifacts does not suffer too much from switching to incremental compilation (see below).

## Data on runtime performance of incrementally compiled release artifacts

Apart from anectodal evidence that runtime performance is "roughly the same" there have been two attempts to measure this in a more reliable way:

1. PR #56678 did an experiment where we compiled the compiler itself incrementally and then tested how the compiler's runtime performance was affected by this. The results are twofold:
    1. In general performance drops by **1-2%** ([compare results](https://perf.rust-lang.org/compare.html?start=3a3121337122637fa11f0e5d42aec67551e8c125&end=26f96e5eea2d6d088fd20ebc14dc90bdf123e4a1) for `clean` builds)
    2. For two of the small test cases (`helloworld`, `unify-linearly`) performance drops by 30%. It is known that these test cases are very sensitive to LLVM making the right inlining decisions, which we already saw when switching from single-CGU to non-incremental ThinLTO. This is indicative that microbenchmarks may see performance drops unless the author of the benchmark takes care of marking bottleneck functions with `#[inline]`.
2. For a limited period of time we made incremental compilation the default in Cargo (https://github.com/rust-lang/cargo/pull/6564) in order to see how this affected measurements on [lolbench.rs](https://lolbench.rs). It is not yet clear if the experiment succeeded and how much useful data it collected since we had to cut it short because of a regression (#57947). The initial data looks promising: only a handful of the ~600 benchmarks showed performance losses (see https://lolbench.rs/#nightly-2019-01-27). But we need further investigation on how reliable the results are. We might also want to re-run the experiment since the regression can easily be avoided.

One more experiment we should do is compiling Firefox because it is a large Rust codebase with an excellent benchmarking infrastructure (cc @nnethercote). 

cc @rust-lang/core @rust-lang/cargo @rust-lang/compiler