Repository: rust
Issue: The `extern "C"` ABI of SIMD vector types depends on target features (tracking issue for `abi_unsupported_vector_types` future-incompatibility lint)
Labels: T-compiler,I-unsound,A-SIMD,C-tracking-issue,F-simd_ffi,A-ABI,A-target-feature,T-opsem,L-abi_unsupported_vector_types

The following program has UB, for very surprising reasons:
```rust
use std::mem::transmute;
#[cfg(target_arch = "x86")]
use std::arch::x86::*;
#[cfg(target_arch = "x86_64")]
use std::arch::x86_64::*;

extern "C" fn no_target_feature(_dummy: f32, x: __m256) {
    let val = unsafe { transmute::<_, [u32; 8]>(x) };
    dbg!(val);
}

#[target_feature(enable = "avx")]
unsafe fn with_target_feature(x: __m256) {
  // Here we're seemingly just calling a safe function... but actually this call is UB!
  // Caller and callee do not agree on the ABI; specifically they disagree on how the
  // `__m256` argument gets passed.
  no_target_feature(0.0, x);
}

fn main() {
    assert!(is_x86_feature_detected!("avx"));
    // SAFETY: we checked that the `avx` feature is present.
    unsafe {
        with_target_feature(transmute([1; 8]));
    }
}
```
The reason is that the ABI of the `__m256` type depends on the set of target features, so the caller (`with_target_feature`) and callee (`no_target_feature`) do not agree on how the argument should be passed. The result is a [vector half-filled with junk](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=ee15f570ee5df1ecbec6321c44066008). (The same issue also arises in the other direction, where the caller has fewer features than the callee: [example here](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=78a91a7266b7e39fe539c1e8d8c9892d).)

Currently, we have no good way to do correct code generation here. See https://github.com/rust-lang/rust/issues/132865 for a discussion of how we could support such code in the future; it will require some non-trivial work. So instead, the current plan is to reject such code entirely.

This is the tracking issue for the lint that moves us in that direction. The hope is that passing SIMD vectors across a C ABI is sufficiently rare, and most of the cases being rejected have anyway already been broken, that we can reject this without much of an ecosystem impact. Crater showed no regressions. 

This is linted against since Rust 1.84, and shown in future breakage reports starting with Rust 1.85. The plan is to move to a hard error with Rust 1.87. 

**When this becomes a hard error:**

- We can probably remove the error tracked at https://github.com/rust-lang/rust/issues/63068?
- And also the warning triggered [in this example](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=10c712ef153066a8c1829f6c7f41a3f4)?

----

**Original issue text:**

I'm not sure if this is currently even properly documented? We are mentioning it in https://github.com/rust-lang/rust/pull/115476 but the program above doesn't involve any function pointers so we really cannot expect people to be aware of that part of the docs. We show a general warning about the type not being FFI-compatible, but that warning shows up a lot and anyway in this case both caller and callee are Rust functions!

I think we need to do better here, but backwards compatibility might make that hard. @chorman0773 suggested we should just reject functions like `no_target_feature` that take an AVX type by-val without having declared the AVX feature. That seems reasonable; a crater run would be needed to assess whether it breaks too much code. An alternative might be to have a deny-by-default lint that very clearly explains what is happening.

There are also details to work out wrt what exactly the lint should check. Newtypes around `__m256` will have the same problem. What about other larger types that contain `__m256`? Behind a ptr indirection it's obviously fine, but what about `(__m256, __m256)`? If we apply the `ScalarPair` optimization this will be passed in registers even on x86.

A possible place to put the check could be [somewhere around here](https://github.com/rust-lang/rust/blob/8fa7bdf191d8658e5ab670766ac0b2d40c7d83c2/compiler/rustc_codegen_ssa/src/codegen_attrs.rs#L279-L309).

In terms of process, I am not sure if an RFC is required; a t-compiler MCP might be sufficient. Currently we accept code that clearly doesn't do what it looks like it should do.

And finally -- are there any other targets (besides x86 and x86-64) that have target-features that affect the ABI? They should get the same treatment.

Note that this is different from https://github.com/rust-lang/rust/issues/116344 in two ways:
- This issue only affects non-"Rust" ABIs; the other issue affects even the "Rust" ABI. (The Rust ABI passes `__m256` by-pointer exactly to work-around this issue.)
- This issue comes about when the user enables extra features (which is possible also locally via `#[target_feature]`), the other issue comes about when the user *disables* features (which is only possible on a per-crate level via `-C`, and if you're mixing crates with different `-C` flags then you're already already on very shaky grounds -- we do that with std but nobody else really gets to do that, I think).

Cc @workingjubilee more ABI fun ;)