Repository: rust
Issue: Massive performance regression between `nightly-2022-08-12` and `nightly-2022-08-13`
Labels: A-LLVM,I-slow,P-medium,T-compiler,regression-from-stable-to-stable,C-bug,E-needs-mcve

Usually I develop on the `stable` channel and wanted to see how [my project](https://github.com/paritytech/wasmi) performs on the upcoming `beta` or `nightly` channels. I saw [performance regressions between 30-80%](https://github.com/paritytech/wasmi/pull/507#issuecomment-1275152300) across the board in native and Wasm targets. Those regressions were confirmed by our benchmarking CI as can be seen by the link.

## Was it the LLVM 15 Update?

I conducted a bisect and found that the change happened between `nightly-2022-08-12` and `nightly-2022-08-13`.
After short research I saw that Rust updated from LLVM 14 to 15 in exactly this time period: https://github.com/rust-lang/rust/pull/99464 Other merged commits in this time period were not as suspicious to me.

## Past Regressions

Also this unfortunately [is not the first time we saw such massive regressions ...](https://github.com/rust-lang/rust/issues/95409).
It is extremely hard to craft a minimal code snippet out of `wasmi` since it is a very heavily optimized bunch of code with lots of interdependencies.
Unfortunately [the `wasmi` project](https://github.com/paritytech/wasmi) is incredibly performance critical to us. Even 10-15% performance regression are a disaster to us let alone those 30-80% we just saw ...

## Hint for Minimal Code Example

**I have one major suspicion:** Due to [missing guaranteed tail calls in Rust](https://github.com/rust-lang/rfcs/issues/2691) we are heavily reliant on a non-guaranteed optimization for our `loop-switch` based interpreter hot path that pulls jumps to the match arms which results to very similar code as what threaded-code interpreter code would produce. The code that depends on this particular optimization [can be found here](https://github.com/paritytech/wasmi/blob/21e12da67a765c8c8b8a62595d2c9d21e1fa2ef6/crates/wasmi/src/engine/executor.rs#L84).
This suspicion is underlined by the fact that especially non call-intense workloads show most regressions in the linked benchmarks. This implies to me that the regressions have something to do with instruction dispatch.

## Potential Future Solutions

- The Rust compiler could add a few benchmarks concerning those `loop-switch` optimizations to [its set of benchmarks](https://github.com/rust-lang/rustc-perf/) so that future LLVM updates won't invalidate those optimizations. I am not sure how viable this approach is to the Rust compiler developers though. Also this only works if we find all the fragile parts that cause these regressions.
- Ideally Rust offered abstractions that allow to develop efficient interpreters in Rust without relying on Rust/LLVM optimizations: for example guaranteed tail calls.

## Reproduce

The current `stable` Rust channel is the following:

```
stable-x86_64-unknown-linux-gnu (default)
rustc 1.64.0 (a55dd71d5 2022-09-19)
```

In order to reproduce these benchmarks do the following:

1) `git clone git@github.com:paritytech/wasmi.git`
1) `cd wasmi`
1) `git checkout 21e12da67a765c8c8b8a62595d2c9d21e1fa2ef6`
1) `rustup toolchain install nightly-2022-08-12`
1) `rustup toolchain install nightly-2022-08-13`
1) `git submodule update --init --recursive`
1) `cargo +stable bench --bench benches execute -- --save-baseline stable`
1) `cargo +nightly-2022-08-12 bench --bench benches execute -- --baseline stable`
1) `cargo +nightly-2022-08-13 bench --bench benches execute -- --baseline stable`