Repository: rust
Issue: yarte_lexer 0.0.1 compilation regression
Labels: P-medium,regression-from-stable-to-stable,C-bug

@lqd pointed out a [failure](https://crater-reports.s3.amazonaws.com/beta-1.61-2/beta-2022-04-18/reg/yarte_lexer-0.0.1/log.txt) from a recent crater run.

Bisection identifies the 7th commit of #95159 as the problem: "[Eliminate TokenTreeOrTokenTreeSlice](https://github.com/rust-lang/rust/pull/95159/commits/31df6807893bd9c66cba57ff0f4de89ab9d8460e)". Interestingly, #95797 reverted many of the changes from that 7th commit, but the problem persists.

**UPDATE: See [this comment](https://github.com/rust-lang/rust/issues/96305#issuecomment-1106005128) for the explanation.**

I have a reduced test case. It doesn't fit in a single file, instead requiring two crates.

## Two-crate test case

Crate `y`, `Cargo.toml`:
```toml
[package]
name = "y"
version = "0.1.0"
edition = "2018"
[dependencies]
```
Crate `y`, `src/lib.rs`:
```rust
#![feature(trace_macros)]
trace_macros!(true);

macro_rules! ascii_builder {
    ($($n:literal)+) => {
        #[macro_export]
        macro_rules! ascii {
            $(($n) => { $n };)+
            ($t:tt) => { compile_error!("no match") };
        }
    };
}

ascii_builder!('a' 'b' 'c');
```
Crate `z` (which sits next to crate `y` in the filesystem), `Cargo.toml`:
```toml
[package]
name = "z2"
version = "0.1.0"
edition = "2021"
[dependencies]
y = { path = "../y" }
```
Crate `z`, `src/main.rs`:
```rust
#![feature(trace_macros)]
trace_macros!(true);

fn main() {
    let _x = y::ascii!('a');
}
```
Note: The original code defines `ascii` and `ascii_builder` within the `yarte_lexer` crate and then uses `ascii` within a unit test. For my reduced test case I used two separate crates because it made it easier to use tools like `cargo expand`, and to avoid any issues specific to unit tests.

## Output
If I compile with `cargo check` using a version prior to #95159, compilation succeeds with this output:
```
    Checking y v0.1.0 (/home/njn/dev/y)
note: trace_macro
  --> /home/njn/dev/y/src/lib.rs:14:1
   |
14 | ascii_builder!('a' 'b' 'c');
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: expanding `ascii_builder! { 'a' 'b' 'c' }`
   = note: to `#[macro_export] macro_rules! ascii
           {
               ('a') => { 'a' } ; ('b') => { 'b' } ; ('c') => { 'c' } ; ($t : tt) =>
               { compile_error! ("no match") } ;
           }`

    Checking z2 v0.1.0 (/home/njn/dev/z3)
note: trace_macro
 --> src/main.rs:5:14
  |
5 |     let _x = y::ascii!('a');
  |              ^^^^^^^^^^^^^^
  |
  = note: expanding `ascii! { 'a' }`
  = note: to `'a'`

    Finished dev [unoptimized + debuginfo] target(s) in 0.21s
```
If I do the same with a version after #95159, compilation fails with this output:
```
    Checking y v0.1.0 (/home/njn/dev/y)
note: trace_macro
  --> /home/njn/dev/y/src/lib.rs:14:1
   |
14 | ascii_builder!('a' 'b' 'c');
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: expanding `ascii_builder! { 'a' 'b' 'c' }`
   = note: to `#[macro_export] macro_rules! ascii
           {
               ('a') => { 'a' } ; ('b') => { 'b' } ; ('c') => { 'c' } ; ($t : tt) =>
               { compile_error! ("no match") } ;
           }`

    Checking z2 v0.1.0 (/home/njn/dev/z2)
error: no match
 --> src/main.rs:5:14
  |
5 |     let _x = y::ascii!('a');
  |              ^^^^^^^^^^^^^^
  |
  = note: this error originates in the macro `y::ascii` (in Nightly builds, run with -Z macro-backtrace for more info)

note: trace_macro
 --> src/main.rs:5:14
  |
5 |     let _x = y::ascii!('a');
  |              ^^^^^^^^^^^^^^
  |
  = note: expanding `ascii! { 'a' }`
  = note: to `compile_error! ("no match")`

error: could not compile `z2` due to previous error
```
The obvious difference is the compile error. Interestingly, the expansion of `ascii_builder` in crate `y` is the same in both cases. So it's not at all clear why the failure occurs. There's a rule matching `'a'` in both cases.

## Single file attempt
I tried and failed to reproduce this in a single file. Here's what I had:
```rust
macro_rules! ascii_builder {
    ($($n:literal)+) => {
        #[macro_export]
        macro_rules! ascii {
            $(($n) => { $n };)+
            ($t:tt) => { compile_error!("no match") };
        }
    };
}

ascii_builder!('a' 'b' 'c');

fn main() {
    let _x = ascii!('a');
}
```
I just get the same compile error from above, no matter which version of the compiler I use:
```
error: no match
  --> y.rs:9:26
   |
9  |             ($t:tt) => { compile_error!("no match") };
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
...
17 |     let _x = ascii!('a');
   |              ----------- in this macro invocation
   |
   = note: this error originates in the macro `ascii` (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error
```
I honestly don't know why this single-file version fails but the two-crate version works, and whether that is valid or a bug. It feels like it should compile.

## Metadata difference
The only other clue I have is that the metadata has changed slightly. The older (pre-regression) compiler produces this:
```
-rw-rw-r-- 1 njn njn 3282 Apr 22 13:39 liby-d5c1be4b71acf3c4.rmeta
```
The newer (post-regression) compiler produces this:
```
-rw-rw-r-- 1 njn njn 3278 Apr 22 13:39 liby-d5c1be4b71acf3c4.rmeta
```
I don't know of an easy way to compare metadata files, so I don't know what accounts for the slight size difference.

## Conclusion
This is a weird one, involving a macro that generates another macro that is then used in another crate. This explains why the change in behaviour took a while to be noticed.

I have looked through the commit responsible, but I can't see any changes related to all of the above.

@petrochenkov, any ideas?