Repository: rust
Issue: Call stack exhaustion (overflow) in parser with a very large generated file
Labels: I-ICE,A-parser,T-compiler,C-bug,A-cranelift

<!--
Thank you for finding an Internal Compiler Error! 🧊  If possible, try to provide
a minimal verifiable example. You can read "Rust Bug Minimization Patterns" for
how to create smaller examples.
http://blog.pnkfx.org/blog/2019/11/18/rust-bug-minimization-patterns/
-->

### Code

Can't provide a minimal example for obvious reasons :)

```
$ git clone https://github.com/bytecodealliance/wasmtime/
$ cd wasmtime
$ git checkout 87817f38a128caa76eaa6a3c3c8ceac81a329a3e
$ cd cranelift/codegen
$ RUST_MIN_STACK=1048576 cargo +1.80.0 build
```

On my Linux x86_64 system this deterministically crashes in the rustc parser due to call stack exhaustion (i.e. a call stack overflow). The full output is [here](https://gist.github.com/sunshowers/3ac000e5a5022acd3f07886a16a39520). (Note I haven't built a Rust compiler with symbols, because I have enough information to establish the cause without needing symbols.)

The file that's failing to parse is [this one](https://gist.githubusercontent.com/sunshowers/22b1a612ee1cb88047c456532e0f6877/raw/9403b8b644f25c600d426ee1a39e2459684244dc/gistfile0.txt
). It is an autogenerated file called `isle_opt.rs`, and is generated by the `cranelift-codegen` build script.

This also reproduces with `rustc 1.82.0-nightly (f8060d282 2024-07-30)`.

What's the `RUST_MIN_STACK` doing, you might ask? Well, long story behind this, but I got here by first diagnosing the issue on illumos, which crashed in the same spot with the same crate ([illumos stack trace](https://gist.github.com/sunshowers/5edb7207c1e1234b0400bc6517f45b29)), **without requiring `RUST_MIN_STACK` to be set.**

* In fact, on illumos, the issue occurs whether `RUST_MIN_STACK` is set to 1MiB, 2MiB (default) or 4MiB. This is consistent with the description below.

What's happening is:

- This file in `cranelift-codegen` is triggering the crash by requiring more than 1MiB of stack space.
  - The `rustc` parser running against `cranelift-codegen` needs more than 1MiB of stack space, but less than 2MiB.
- Had this bug occurred on other platforms like Linux, this issue would have been a showstopper. However, it wasn't visible on those platforms because:
  - Threads created by Rust use a 2MiB stack by default.
  - `rustc` requests a [1MiB stack segment with a 100KiB red zone](https://github.com/rust-lang/rust/blob/dba8e2d2c2890a8b9e88cbf4855ac5711337946c/compiler/rustc_data_structures/src/stack.rs#L17).
  - On other platforms like Linux, `stacker` can see that well over 100KiB of stack space is left, and so it does not allocate a new segment.
  - On illumos, [this logic doesn't exist](https://github.com/rust-lang/stacker/blob/5df2309ccf7b1671909386c2670c7342a4d44142/src/lib.rs#L412-L457). `stacker` cannot see how much stack was left, and so it unconditionally allocates a new 1MiB segment.
  - This 1MiB stack is simply not enough to parse `isle_opt.rs`.

With `RUST_MIN_STACK` used to set a stack size <= 1MiB, we would expect that:

1. `rustc` calls `stacker` as before.
2. There are two possibilities: either `stacker` decides there is enough stack space and doesn't create a new segment, or it decides there isn't enough and does create a new 1MiB segment.
3. In either case, 1MiB is simply not enough to parse `cranelift-codegen`, and the program crashes.

To the best of my understanding, **this is a bug in rustc's use of stacker.** The fact that 1MiB just isn't enough to parse that file was being masked by the default stack size of 2MiB.

I think the fix is that **`rustc` should be calling `stacker` more often in its recursive sections** -- if it did, then stacker would allocate a new segment as soon as less than 100KiB of stack space was available.

(A secondary issue is that stacker should be able to detect stack sizes on illumos -- I'll try sending a PR for that separately.)

### Meta

Reproes with both:

`rustc --version --verbose`:

```
rustc 1.80.0 (051478957 2024-07-21)
binary: rustc
commit-hash: 051478957371ee0084a7c0913941d2a8c4757bb9
commit-date: 2024-07-21
host: x86_64-unknown-linux-gnu
release: 1.80.0
LLVM version: 18.1.7
```

and

```
rustc 1.82.0-nightly (f8060d282 2024-07-30)
binary: rustc
commit-hash: f8060d282d42770fadd73905e3eefb85660d3278
commit-date: 2024-07-30
host: x86_64-unknown-linux-gnu
release: 1.82.0-nightly
LLVM version: 18.1.7
```

### Error output

https://gist.github.com/sunshowers/3ac000e5a5022acd3f07886a16a39520