Repository: godot
Issue: Improve CI experience for contributors
Labels: enhancement,discussion,topic:buildsystem

<!-- Please search existing issues for potential duplicates before filing yours:
https://github.com/godotengine/godot/issues?q=is%3Aissue
-->

**Godot version:**
<!-- Specify commit hash if using non-official build. -->
N/A

**OS/device including version:**
<!-- Specify GPU model, drivers, and the backend (GLES2, GLES3, Vulkan) if graphics-related. -->
N/A

**Issue description:**
<!-- What happened, and what was expected. -->
Getting large PRs to pass CI checks can be overly involved and time consuming. I'm probably the most vocal on this subject (sorry! :grin: ), I suspect some others are too polite to mention, but being an impatient old guy I thought I'd put in some experience from a user perspective and some suggestions and have a rant. They may not all be actionable / best option, and I may well be flat out wrong in some areas.

The problem is mainly that it is currently difficult to run the same checks (that are done by CI) locally. They need not necessarily be *impossible*, just difficult is enough to prevent use (if you have to install docker + a dozen tools it isn't going to happen). So you often get caught in a loop of

1) push PR to github
2) wait for significant amount of time (up to 30-40 mins) for CI to find an error
3) fix error
4) return to step 1

**With a small PR, this isn't a massive problem. But with large PRs, this process simply is not efficient or practical.**

#### Hooks
To try and make this easier we have hooks. But in my experience these involve installing a bunch of other software, which must be the exact correction version in some cases (e.g. clang-format), resolve conflicts with the OS, and get these to run, only to find that the CI may do extra checks that these hooks don't seem to cover (like formatting within comments).

Aaronfranke suggested that the CI produces a patch file, but I have no idea where this is or how to use it (instructions?). On top of that, given that compilation stops on the first error, I don't think it helps much if you have e.g. 1000 errors to work through. Of course you can turn warnings to full locally, but some compilers that you may not have access to (e.g. cl.exe) may produce different warnings to gcc / clang.

#### Tests
On top of the formatting issue, we also now have unit tests and testing via Qarmin's project which is needed to pass CI.

If it doesn't pass, the problem is that this testing is done at the end of CI (sometimes after a ~35 minute wait), and these failing these tests can be difficult to resolve with the limited information provided. The solution I ended up with was to figure out how to run the test project locally, otherwise I would have simply never found the bug in question, given up and chosen a different hobby!

## Some suggestions for improving the process for contributors
### Limit formatting tools
Don't require a long list of formatting tools to be installed for the most common use case, c++ code. Really you should pick a tool and version - e.g. clang-format 10, use the features provided by that tool only. If it doesn't offer such and such feature, to do just the right amount of tabs in such and such a case, simply don't use that feature.

The amount of style checks are now bordering on pedantic. My rule of thumb would be - if you had to explain these checks to a psychiatrist, and he would offer you prozac etc, you are probably going too far.

We as programmers tend to be very OCD (I know I do), but the trick with compulsive behaviour is to notice when it is becoming a problem and that you are being irrational, and reign it in. Style checks should be there to make things easier, not to make things more difficult. At the moment we are in a situation where they create more problems than they solve. So while style checking is good - it is essential to make them practical and easy to use.

Maybe we could cover all cases by a combination of specified version of clang-format, and a custom tool written in c or python or some such so that it was easy for users to install / use. Maybe we can do everything with clang-format, that would be even better.

### Testing
We should have detailed instructions so contributors can download and run tests locally. I worked out from the github action that I could build with sanitizers and use qarmin's test project. But this should be in the documentation for contributors, step by step instructions to do this kind of thing. Treat us as morons, and you won't be far wrong. :grinning: 

### Compilation - warnings as errors
A real issue is that the different compilers in CI given different warnings, and thus errors. So a clean run on your own machine doesn't mean you'll pass CI. One possibility is to limit the CI warnings as errors check to one compiler / version / platform only. This means that, provided everyone on windows and linux could install that compiler, everyone could potentially guarantee to pass CI checks locally.

Warnings could still be important on another platform so this is not ideal. Another option is to compile without warnings as errors (thus creating a long list of issues that could be fixed in one go), instead of terminating the compile on the first error. You could then search string through the compilation output for a warning string, and use that to signify the CI had failed, instead of the current method.

### Rule of Thumb
Overall, if I was deciding on such a system, the rule would be, if the check can't be reproduced simply and easily on contributors machines, it shouldn't be required to pass CI. This may not be completely practical in all cases, but the gist is good.

Anyway these are just some suggestions and viewpoints, hopefully to get some discussion / positive changes made.