Repository: rust
Issue: Tracking Issue for consolidating command invocations in bootstrap
Labels: T-bootstrap,C-tracking-issue

This is a tracking issue for monitoring the work of improving how we deal with command invocations in bootstrap (proposed [here](
https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap)).

Currently, bootstrap executes commands on many places, often in arbitrary ways. This makes it harder to debug, profile and test the command invocations executed by it. The goal of this refactoring is to make sure that all external command invocations use the same API (a custom command wrapper) and that they go through a centralized location.

Intermediate steps:
- [x] Introduce a command wrapper abstraction
  - https://github.com/rust-lang/rust/pull/116581
- [x] Port all existing simple command usages that have access to `&Build(er)` to `BootstrapCommand`
  - https://github.com/rust-lang/rust/pull/126731
  - https://github.com/rust-lang/rust/pull/126822
  - https://github.com/rust-lang/rust/pull/127120
- [x] Modify `BootstrapCmd` so that it stores `Command` instead of `&mut Command` and remove all calls to `BootstrapCmd::from` by changing `Command::new` to `BootstrapCmd::new`
  - https://github.com/rust-lang/rust/pull/126822
- [x] Consolidate how command output modes work (described [here](https://github.com/rust-lang/rust/pull/126731))
  - https://github.com/rust-lang/rust/pull/127120
- [x] Make it easier to use the command API in a fluent way, and mark methods with `#[must_use]`
  - https://github.com/rust-lang/rust/pull/127321
- [x] Add `#[track_caller]` to command execution functions to make debugging easier.
  - https://github.com/rust-lang/rust/pull/127450
- [x] Implement drop bombs to check that each command has been executed at least once.
  - https://github.com/rust-lang/rust/pull/127450
- [x] Refactor the rest of command executions not currently using `BootstrapCmd` that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
  - https://github.com/rust-lang/rust/pull/127680 
- [x] Make the API for getting a stdout of a command nicer.
  - https://github.com/rust-lang/rust/pull/127799
- [x] Add a check that asserts that you don't try to get stdout/stderr of a command that does not capture stdout/stderr.
  - https://github.com/rust-lang/rust/pull/127799
- [x] Refactor how flags are passed to `Config`.
  - https://github.com/rust-lang/rust/pull/128878
- [ ] Remove access to inner command of `BootstrapCommand`.  
- [ ] Handle the most complex cases, such as output streaming.
- [ ] Handle the co-operation of `Builder`, `Build`, `Config` and command context. There are some fields and logic used during command execution that are distributed amongst `Builder/Build/Config`, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
- [ ] Refactor the rest of commands that cannot access builder (e.g. `Config::parse`) by introducing a new command context that will be passed to these places, and then stored in `Builder`. Move certain fields (such as `fail_fast`) from `Builder` to the command context.
- [ ] Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the `Step` debug hierarchical output.
- [ ] Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to `metrics.json`).
- [ ] Implement caching of commands.
- [ ] Implement testing of commands through snapshot tests/mocking.