Repository: rust
Issue: Latest parallel compiler ICEs on large crates - attempt to steal from stolen value
Labels: I-ICE,T-compiler,C-bug,A-parallel-queries,WG-compiler-parallel

### Meta
[Self-compiled](https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html) latest (5dab47dcd8267b8769421b46532414ec36d625e3) Rust [with parallel compilation enabled](https://rustc-dev-guide.rust-lang.org/parallel-rustc.html):

`config.toml`:
```toml
# Includes one of the default files in src/bootstrap/defaults
profile = "user"
changelog-seen = 2
[rust]
parallel-compiler = true
[llvm]
download-ci-llvm = true
```

<details><summary>`./x.py build --stage 2 compiler/rustc`</summary><p>

```
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage3 compiler (x86_64-unknown-linux-gnu)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/home/ten/Projects/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/panicking.rs:101:14
   2: bootstrap::Build::read_stamp_file
             at ./src/bootstrap/lib.rs:1296:24
   3: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at ./src/bootstrap/compile.rs:1107:27
   4: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1574:23
   5: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:627:9
   6: <bootstrap::compile::Assemble as bootstrap::builder::Step>::make_run
             at ./src/bootstrap/compile.rs:1034:30
   7: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:175:13
   8: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:231:21
   9: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:619:9
  10: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:599:9
  11: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:624:13
  12: bootstrap::main
             at ./src/bootstrap/bin/main.rs:33:5
  13: core::ops::function::FnOnce::call_once
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Build completed unsuccessfully in 0:03:53
```

</p></details>

`rustc +stage2 --version --verbose`:
```
rustc 1.58.0-dev
binary: rustc
commit-hash: unknown (but actually 5dab47dcd8267b8769421b46532414ec36d625e3)
commit-date: unknown (but actually Mon Oct 18 06:46:03 2021 +0000)
host: x86_64-unknown-linux-gnu
release: 1.58.0-dev
LLVM version: 13.0.0
```

### Code

Unfortunately, I didn't manage to reproduce this consistently because the issue appears to be non-deterministic and didn't happen when trying to reproduce it on a minimal project.
Best I can recommend right now is to try to compile a _large project_ with `-Z threads=n` with `n` > 1, which causes this on diverse amount of crates, but doesn't happen with `n` = 1, nor without `-Zthreads=n` set in RUSTFLAGS.

### Error output
```bash
RUSTFLAGS="-Zthreads=10" cargo +stage2 check
```
```
thread 'rustc' panicked at 'attempt to steal from stolen value', compiler/rustc_mir_transform/src/lib.rs:622:33
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_display
   3: core::option::expect_failed
   4: rustc_mir_transform::promoted_mir
   5: core::ops::function::FnOnce::call_once
   6: rustc_rayon_core::tlv::with
   7: rustc_query_system::query::plumbing::try_execute_query
   8: rustc_query_system::query::plumbing::get_query
   9: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
  10: rustc_rayon::iter::plumbing::bridge_producer_consumer::helper
  11: rustc_rayon_core::registry::in_worker
  12: rustc_rayon::iter::plumbing::bridge_producer_consumer::helper
  13: rustc_rayon_core::job::StackJob<L,F,R>::run_inline
  14: rustc_rayon_core::registry::in_worker
  15: rustc_rayon::iter::plumbing::bridge_producer_consumer::helper
  16: rustc_rayon_core::registry::in_worker
  17: rustc_rayon::iter::plumbing::bridge_producer_consumer::helper
  18: rustc_rayon_core::registry::in_worker
  19: rustc_rayon::iter::plumbing::bridge_producer_consumer::helper
  20: <rustc_rayon::vec::IntoIter<T> as rustc_rayon::iter::IndexedParallelIterator>::with_producer
  21: rustc_metadata::rmeta::encoder::prefetch_mir
  22: rustc_rayon_core::registry::in_worker
  23: <rustc_rayon_core::job::StackJob<L,F,R> as rustc_rayon_core::job::Job>::execute
  24: rustc_rayon_core::registry::WorkerThread::wait_until_cold
  25: rustc_rayon_core::registry::ThreadBuilder::run
  26: scoped_tls::ScopedKey<T>::set
  27: crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.58.0-dev running on x86_64-unknown-linux-gnu

note: compiler flags: -Z threads=10 -C embed-bitcode=no -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [promoted_mir] optimizing promoted MIR for `token::<impl at /home/ten/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-0.15.44/src/token.rs:281:13: 283:14>::hash`
end of query stack
error: build failed
```

The `query stack` and backtrace change over runs, but the end error stays the same, e.g.:
```
thread 'rustc' panicked at 'attempt to steal from stolen value', compiler/rustc_mir_transform/src/lib.rs:622:33
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_display
   3: core::option::expect_failed
   4: rustc_mir_transform::promoted_mir
   5: core::ops::function::FnOnce::call_once
   6: rustc_rayon_core::tlv::with
   7: rustc_query_system::query::plumbing::try_execute_query
   8: rustc_query_system::query::plumbing::get_query
   9: rustc_metadata::rmeta::encoder::EncodeContext::encode_crate_root
  10: rustc_metadata::rmeta::encoder::encode_metadata_impl
  11: rustc_rayon_core::registry::in_worker
  12: rustc_metadata::rmeta::encoder::encode_metadata
  13: rustc_interface::passes::start_codegen
  14: rustc_rayon_core::tlv::with
  15: rustc_interface::queries::Queries::ongoing_codegen
  16: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  17: rustc_span::with_source_map
  18: rustc_interface::interface::create_compiler_and_run
  19: std::panicking::try
  20: <rustc_rayon_core::job::StackJob<L,F,R> as rustc_rayon_core::job::Job>::execute
  21: rustc_rayon_core::registry::WorkerThread::wait_until_cold
  22: rustc_rayon_core::registry::ThreadBuilder::run
  23: scoped_tls::ScopedKey<T>::set
  24: crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}}
```

# Context

I have a crate that spends one minute in each cargo check in:
https://github.com/rust-lang/rust/blob/b27661eb33c74cb514dba059b47d86b6582ac1c2/compiler/rustc_typeck/src/check/mod.rs#L920-L922

This makes getting feedback in the IDE super slow. However I'm coding on a machine with a ThreadRipper 3970X so if I thought if I used the parallel-enabled compiler just for the cargo check in my IDE I might bring that down to just a few seconds for getting feedback (and still use the regular stable compiler when the binary is required).
Hence this attempt.

That being said it wouldn't be so slow if it didn't attempt to re-typecheck every body, even those that weren't touched.
Not sure what's the blocker to this, but I'd love to see some reference on that topic if anyone has any.