Repository: rust
Issue: Switch `run-make` tests from Makefiles to rust
Labels: A-testsuite,E-hard,C-enhancement,E-help-wanted,T-bootstrap,T-infra,A-compiletest,A-run-make

Rust currently has a suite of [`run-make`](https://github.com/rust-lang/rust/tree/master/src/test/run-make) tests, which generally test specific `rustc` invocations or behaviour,  or require external tools (eg. `grep` or `nm`).

The goal of this issue is to rewrite these `run-make` tests, which are currently written as Makefiles, in rust to a) get rid of the dependency on external tools such as `make` and b) make them more accessible to rust contributors by not requiring arcane knowledge of the `make` tool.

The transition will require at least the following steps:

* [ ] Survey the existing `run-make` tests with regard to what they actually do / which programs they use (and how). I assume the programs will likely fall into one of three categories:
    * Rust tools (`rustc`, `rustdoc`)
    * Utilities easily replaceable by rust code, eg. `grep`
    * Complex utilities, eg. `nm`, which cannot be easily rewritten in rust
* [ ] Design and (partially) implement a support library, which makes the actions identified above easily possible. For example I imagine a test should look something like this:
    ```rust
    extern crate support;
    
    fn main() {
        let s = support::init();
        
        let lib = s.rustc().compile("file1.rs").output_rlib();
        
        assert!(s.nm(lib).filter_lines("some_symbol").count() == 2);
    }
    ```
* [ ] Add a new mode to [`compile-test`](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest), maybe `run-rmake`, which runs the new rust-based `run-make` tests. This will either include compiling the support library, or receiving the support library from a previous build stage.
* [ ] Start porting the actual `run-make` tests, which may involve adding additional functionality to the support library. At that time this issue, or another, will track the state of all the existing tests and include some detailed instruction to allow people to easily contributing by porting one of the existing tests.

---

There are some open questions:

* Should the support library allow execution of arbitrary commands? Limiting commands to only those explicitly added to the support library would mean that there is a single place which lists the external tools we depend on.
* Is the proposed integration with `compiletest` the correct choice? For example the support library itself may have external dependencies  (maybe `regex` or `gcc-rs`) which means we should probably use cargo to compile the actual tests. At which point it may be worth considering if `compile-test` is needed at all or if cargo is enough (Have the support library in `src/lib.rs`,  the new `run-make` tests in `tests/*.rs` and auxiliary files in subdirectories of `tests/` named after the main test file).

---

If anyone wants to get involved in the process, please leave a comment on this issue or ping me on IRC.

---

<details><summary>Original Issue Description</summary>

Based on a short experiment, it looks like rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make, where make is only used for the `run-make` tests as far as I can tell.

Of those five, the first four are easily installable natively on windows, whereas make wasn't as straight forward to installe when I tried and required msys2 / mingw.

The questions the are:

* Is there any interest in performing such a conversion?
* If so, then which language should those tests be migrated to? Python or Rust seem like the logical choices.
* How should the switch happen? We'll probably want some kind of incremental strategy, since there are quite a few `run-make` tests.

</details>