Repository: rust
Issue: [aarch64] Running unit tests with codegen-units=1 triggers errors when optimizing with vectorization
Labels: A-LLVM,T-compiler,C-bug,O-AArch64,C-optimization

When adding `codegen-units=1` in the [firecracker](https://github.com/firecracker-microvm/firecracker) project we saw that some unit tests started failing on `aarch64` with rust 1.52.1.

Upon further investigation, we identified [this loop](https://github.com/firecracker-microvm/firecracker/blob/main/src/arch/src/aarch64/regs.rs#L352) behaving differently depending on different compiler flags - details below.

# Steps to reproduce

1. Check out the branch that enables codegen-units=1 from https://github.com/gc-plp/firecracker/tree/codegen: `git clone -b codegen https://github.com/gc-plp/firecracker`
2. Run `sudo ./tools/devtool shell -p` to launch a debug shell - firecracker uses a container to build binaries.
3. To trigger the unit test failure, run `RUST_TEST_THREADS=1 RUST_BACKTRACE=1 RUSTFLAGS="-D warnings -C link-arg=-lgcc -C link-arg=-lfdt " cargo test --release --target aarch64-unknown-linux-musl  --all --no-fail-fast` in the newly launched shell. This command is the verbose version of what is being executed in the integration test suite [here](https://github.com/firecracker-microvm/firecracker/blob/main/tests/integration_tests/build/test_unittests.py#L20).
4. `vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state` will hang.
5. Run only that test that's hanging: `RUST_TEST_THREADS=1 RUST_BACKTRACE=1 RUSTFLAGS="-D warnings -C link-arg=-lgcc -C link-arg=-lfdt " cargo test --release --target aarch64-unknown-linux-musl  --all --no-fail-fast -- vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state`
6. Will trigger: `process didn't exit successfully: `/firecracker/build/cargo_target/aarch64-unknown-linux-musl/release/deps/vmm-7a653c7da8c736da 'vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state'` (signal: 11, SIGSEGV: invalid memory reference)` 

Expected result is to have the unit test pass as it happens without `codegen-units=1`.

## Additional details

### When the test fails, a coredump file is generated, showing that:
```
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000532630 in arch::aarch64::regs::save_core_registers ()
``` 

### Adding `opt-level = 0` in Cargo.toml and rerunning, the test will pass:
```
running 1 test
test vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state ... ok
```

### Adding `opt-level = 1` in Cargo.toml and rerunning will fail:
```
test vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state ... FAILED

failures:

---- vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state stdout ----
thread 'main' panicked at 'Cannot save state of vcpu: SaveState(GetCoreRegister(Error(2), "FP_VREG0"))', src/vmm/src/vstate/vcpu/aarch64.rs:292:39
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/option.rs:1329:5
   3: vmm::vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state
   4: core::ops::function::FnOnce::call_once
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

### Adding `opt-level = "s"` in Cargo.toml and rerunning will fail:
```
Caused by:
  process didn't exit successfully: `/firecracker/build/cargo_target/aarch64-unknown-linux-musl/release/deps/vmm-47892dfea4706667 'vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state'` (signal: 11, SIGSEGV: invalid memory reference)
```

### Adding `opt-level = "z"` in Cargo.toml and rerunning will pass:
```
running 1 test
test vstate::vcpu::aarch64::tests::test_vcpu_save_restore_state ... ok
```

### Adding `println!("{}", i);` at line 353 in `./src/arch/src/aarch64/regs.rs` will:
- with opt-level=1 it passes
- with opt-level=3 it fails

### Adding `println!("{:x}", id);` at line 354 in `./src/arch/src/aarch64/regs.rs` will:
- With opt-level=1 the test fails: 
`6040000000100054 6040000000100058 604000000010005c 6040000000100060 6040000000100064 6040000000100068 604000000010006c 6040000000100070 6040000000100074 6040000000100078 604000000010007c 6040000000100080 6040000000100084 6040000000100088 604000000010008c 6040000000100090 6040000000100094 6040000000100098 604000000010009c 60400000001000a0 60400000001000a4 60400000001000a8 60400000001000ac 60400000001000b0 60400000001000b4 60400000001000b8 60400000001000bc 60400000001000c0 60400000001000c4 60400000001000c8 60400000001000cc 60400000001000d0 60400000001000d4`

- With opt-level=0 the test passes: 
`6040000000100054 6040000000100058 604000000010005c 6040000000100060 6040000000100064 6040000000100068 604000000010006c 6040000000100070 6040000000100074 6040000000100078 604000000010007c 6040000000100080 6040000000100084 6040000000100088 604000000010008c 6040000000100090 6040000000100094 6040000000100098 604000000010009c 60400000001000a0 60400000001000a4 60400000001000a8 60400000001000ac 60400000001000b0 60400000001000b4 60400000001000b8 60400000001000bc 60400000001000c0 60400000001000c4 60400000001000c8 60400000001000cc 60400000001000d0`

Notice the extra `60400000001000d4` printed with opt-level=1.

### Upgrading to Rust 1.54 will make the test pass

***

Questions:
- Could this be an LLVM issue? The MIR representation doesn't seem to change a lot when instrumenting the loop with the extra print statements. Both rust 1.52.1 and 1.54 seem to use LLVM 12, so there shouldn't be major differences, right?
- Any idea if someone has stumbled upon this before?