Repository: rust
Issue: Tracking Issue for assorted compiletest maintenance
Labels: C-cleanup,T-compiler,T-bootstrap,C-tracking-issue,A-compiletest

This is a tracking issue for a series of `compiletest` cleanups. This tracking issue is on-going and will be edited as suitable to reflect the next steps. Tasks should be broken up into small actionable items.

### Motivation

Currently `compiletest` is a bit of a mess causing it to be really hard to maintain. Let's try to do some housekeeping in `compiletest` to make it easier to maintain.

### Phase 1: `compiletest/src/runtest.rs` cleanups

- [x] Step 1: Break up `compiletest/src/runtest.rs` into smaller helper modules. (https://github.com/rust-lang/rust/pull/130566)
    - Originally https://github.com/rust-lang/rust/issues/89475
- [x] Step 2: Investigate and rework how `valgrind` test suites are handled, namely what happens if `valgrind` is not available.
    - Deleted the `run-pass-valgrind` test suite and valgrind support in #131351.
    - https://github.com/rust-lang/rustc-dev-guide/pull/2091
- [ ] Step 3: Relocate functions on `TestCx` that does not need to be on `TestCx` (especially ones that don't depend on `TestCx` itself) to suitable locations.
- [ ] Step 4: Reorganize methods on `TestCx`:
    - Step 4.1: Privatize methods only used by a specific test suite/mode to their specific helper modules.
    - Step 4.2: Reorder/regroup methods on the core `TestCx` in `runtest.rs` to make it easier to navigate.
- [ ] Step 5: Improve documentation around `runtest.rs`:
    - [x] Step 5.1: Make sure tool docs are registered for compiletest
        - https://github.com/rust-lang/rust/issues/130564
        - https://github.com/rust-lang/rust/pull/130567
    - [ ] Step 5.2: Document util and helper methods on `TestCx`.
    - [ ] Step 5.3: Document individual test suites/modes.
    - [ ] Step 5.4: Document top-level `TestCx` and types/concepts in `runtest.rs`.
    - [ ] Step 5.5: Update `rustc-dev-guide` docs about the individual test modes/suites and about test running.
    - [ ] Step 5.6: Add an example in `rustc-dev-guide` about how to add a new test suite/mode.
- [ ] Step 6: Review implementation of each test suite/mode.

### Phase 2: Rework compiletest error handling and error reporting

- Step 1: Investigate how `compiletest` currently handles errors and reports them.
- Step 2: Come up with a design to make `compiletest` error reporting more cohesive and more helpful for users.
- TODO

### Phase 3: Rework directive handling

- [ ] Step 1: Survey existing directive handling related bugs.
    - [x] Step 1.1: Creating a tracking issue: https://github.com/rust-lang/rust/issues/131425.
    - [ ] Step 1.2: Write a document to describe current limitations/problems with how directives are handled.
- [ ] Step 2: Redesign how directives are parsed and handled.
    - [ ] Step 2.1: Write a document to describe the considerations for directive handling.
    - [ ] Step 2.2: Draft a prototype toy impl for the reworked handling would look like.
    - [ ] Step 2.3: Draft a MCP to propose implementing the rework.
    - @tgross35 has some experimentation in https://github.com/rust-lang/rust/pull/128070, this should get an MCP to receive more feedback from other compiler team members (TODO(@jieyouxu): write-up an MCP)
    - ~~https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Test.20header.20commands.20could.20just.20be.20toml.3F proposes we can use structured TOML directives in doc comments.~~ non-starter as it makes writing tests quite annoying.
- [ ] Step 3: Implement more robust directive handling.
    - [ ] Step 3.1: Implement the more robust design but not merge yet, as we need to...
    - [ ] Step 3.2: ... find out which tests contain invalid directives w.r.t. new directive handling and fix them.
    - [ ] Step 3.3: Investigate and improve testing for directive handling.
    - [ ] Step 3.4: Try to land the improved directive handling.
- [ ] Step 4: Improve directive documentation in source and in rustc-dev-guide:
    - [x] Step 4.1: Unify terminology: stick with "directive", not also "header" or "comment" etc.
        - https://github.com/rust-lang/rustc-dev-guide/pull/2089 
    - [ ] Step 4.2: Document individual directives: syntax, applicable to which test suites, behavior
        - https://github.com/rust-lang/rustc-dev-guide/pull/2089
        - This can be improved once directive handling is more robust
- TODO

There are more phases intended, but they are to be planned.

### Discussions

Rubber-ducking thread: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/.28Rubberducking.29.20compiletest.20test.20discovery.20.2F.20directives