Repository: pytorch
Issue: [feature request] making pytorch less noisy 
Labels: high priority,module: cpp-extensions,feature,triaged,needs design,module: ux

## 🚀 Feature

How can we make pytorch be noise-conscious when it prints warnings.

For some reason when developers write code that includes warnings they at times miss thinking about the big picture and the deployment of the software.

Below you will find multiple situations to consider using a single use-case of `cpp_extension.load` which prints a huge warning and I currently have it printed 1.5K times on a 512 gpus, each print is some 20 lines long.

## 1. repetitive multi-node logs

One of the big issues is that on multi-node the same warning gets printed as many times as there are processes. Which is a huge issue when one has say 512 processes.

Currently with 512 gpus I get this printed 512 times:

```
                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) is not compatible with the compiler Pytorch was
built with for this platform, which is g++ on linux. Please
use g++ to to compile your extension. Alternatively, you may
compile PyTorch from source using c++, and then you can also use
c++ to compile your extension.

See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
with compiling PyTorch from source.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(WRONG_COMPILER_WARNING.format(
/gpfswork/rech/six/commun/conda/tr10-13B/lib/python3.8/site-packages/torch/utils/cpp_extension.py:295: UserWarning: 
```

This is only a part of the log, the 2nd part is:
```
Detected CUDA files, patching ldflags
Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr10-13B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja...
Building extension module scaled_upper_triang_masked_softmax_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module scaled_upper_triang_masked_softmax_cuda...
```
which also gets repeated as many times. But it's of a somewhat different nature, since it's not a warning.

How can we make pytorch be aware of uses in mutli-gpu environment and only print any such advisory warnings on global rank 0?

This is not the only warning, it's just the one I'm dealing with at this moment.

I used a workaround in the application where I shutdown warnings for the non-rank 0 while loading the extensions:

```
    if torch.distributed.get_rank() == 0:
        fused_kernels.load(args)
        torch.distributed.barrier()
    else:
        torch.distributed.barrier()
        with warnings.catch_warnings():
            # ignore loading noise
            warnings.simplefilter("ignore")
            fused_kernels.load(args)
```

Now it prints this on rank 0 only:

## 2. repetitive warnings in the same process

But since Megatron-LM builds a handful of these cuda extensions one after another this warnings still gets printed multiple times.

How can we make the software be aware that it already printed the warnings and that repeating it doesn't add any useful information.

## 3. the warning needs to be printed only when it's relevant. 

The Megatron-LM extensions build just fine despite the warning and yet the warning gets printed every time the application is started regardless of whether it is going to build the extension or not. Surely, this warning is relevant for the actual building of the extension, but if `cpp_extension.load` can see the extension doesn't need to be built why emit the irrelevant warning?

## 4. there needs to be a way to override a warning

Finally, a warning is useful the first 5 times it appears and if the user chooses not to do something about it, either by choice or often because they don't have that choice, there needs to be a way to tell pytorch to no longer emit this warning.

## 5. shutting warnings off is a bad idea

The other approach is for the user to just turn pytorch warnings off, but then they may miss important warnings.

This approach can't be used in frameworks since we can't decide for the user what they want or not.

Practically, as I'm dealing with the same issue in many components, I'm [proposing at HF Transformers](https://github.com/huggingface/transformers/issues/14455) to devise a new type of warnings, which are called with a custom `warning_advise` method and can be turned off via an env var. This is a special deployment mode where the person running the deployment is already aware of all the possible warnings and may choose to turn those off. Note, that these are not all warnings, but a sub-group of advisory warnings only.


Thank you!

--------------

More multi-node repeat / should-be-able-to-shut-off-in-general-advice-type-of-warning candidates:

```
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
```

```
/gpfswork/rech/six/commun/conda/tr10-13B/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run:
```

```
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
```

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @malfet