Repository: rust
Issue: Poor performance with multiple codegen units due to Iterator::fold and FnMut::call_mut inlining failure
Labels: I-slow,T-compiler,C-bug

I am working on a 2D stencil computation (each output element is computed from a weighted sum of NxN input elements, here N = 3), and one of the backends has an innermost loop that looks like this:

```rust
// Compute diffusion gradient
let [full_u, full_v] = (win_u.rows().into_iter())
    .zip(win_v.rows())
    .zip(self.params.weights.0)
    .flat_map(|((u_row, v_row), weights_row)| {
        (u_row.into_iter().copied())
            .zip(v_row.into_iter().copied())
            .zip(weights_row)
    })
    .fold(
        [Values::splat(0.); 2],
        |[acc_u, acc_v], ((stencil_u, stencil_v), weight)| {
            let weight = Values::splat(weight);
            [
                mul_add(weight, stencil_u - u, acc_u),
                mul_add(weight, stencil_v - v, acc_v),
            ]
        },
    );
```

The full computation can be found here : https://github.com/HadrienG2/grayscott .

In case you're wondering why I'm doing the flat_map dance instead of using ndarray's flattened iterator, it's because I get better codegen that way. The reason is probably that otherwise, rustc/LLVM can't figure out that the iteration pattern on the ndarray windows and the fixed-size weights array is the same. Maybe a bug report for another day?

---

If I build it with `codegen-units = 1` as currently configured in the Cargo.toml, I get very satisfactory performance:

```text
$ cargo +nightly criterion -p compute_autovec -- 'compute,2048x.*32'

[...]

compute_autovec/compute,2048x1024elems,32steps                                                                            
                        time:   [79.955 ms 80.610 ms 81.298 ms]
                        thrpt:  [825.47 Melem/s 832.51 Melem/s 839.33 Melem/s]
```

However, if I build with multiple codegen units by commenting out `codegen-units = 1`, performance drops by a factor of 16x.

```text
compute_autovec/compute,2048x1024elems,32steps                                                                          
                        time:   [1.2161 s 1.2168 s 1.2175 s]
                        thrpt:  [55.121 Melem/s 55.151 Melem/s 55.182 Melem/s]
                 change:
                        time:   [+1396.9% +1409.5% +1422.0%] (p = 0.00 < 0.05)
                        thrpt:  [-93.430% -93.375% -93.319%]
                        Performance has regressed.
```

According to a perf analysis, this happens because `<Map as Iterator>::fold()` and `<{closure} as FnMut>::call_mut()` are not correctly inlined in this configuration. On very short inner loops like this, this is unfortunately a deadly mistake for performance.

Please ensure that these two building blocks stay inlined in the presence of multiple codegen units, so that I do not have to pick two between fast builds, good performance, and cool iterator abstractions. See the cargo criterion command above for a reproducer that you should be able to re-run locally.

### Meta

This happens on stable rustc 1.70 and I have reproduced it with the current nightly.

`rustc --version --verbose`:
```
rustc 1.72.0-nightly (f20afcc45 2023-07-04)
binary: rustc
commit-hash: f20afcc455bbcc5c0f7679450fb35fd0c9668880
commit-date: 2023-07-04
host: x86_64-unknown-linux-gnu
release: 1.72.0-nightly
LLVM version: 16.0.5
```