Repository: pytorch
Issue: [RFC] Provide an API for Structural Performance Tips in DDP
Labels: oncall: distributed,feature,triaged,module: ddp

## Motivation

### I. Improving Unstructured and Scattered Performance Tips

Currently a few performance-related tips/warnings w.r.t. DDP are either listed in different sources (e.g., PyTorch Lightning has a dedicated webpage for [performance tips](https://pytorch-lightning.readthedocs.io/en/stable/benchmarking/performance.html
)), or scattered in the log in the form of warnings. As a result, these valuable tips are often 1) not known by the users who have not seen these performance tips, 2) buried by a much larger volume of logs, or 3) noticed by the user but forgotten to be applied.

For example, DDP [reducer.cpp](https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/reducer.cpp) has the following warning to suggest turning off unnecessary `find_unused_parameters` for performance gain.

```
  // Warn user about unnecessary perf hit if all parameters were used in
  // forward.
  if (unused_parameters_.empty()) {
    TORCH_WARN_ONCE(
        "find_unused_parameters=True was specified in DDP constructor, "
        "but did not find any unused parameters in the forward pass. This flag "
        "results in an extra traversal of the autograd graph every iteration, "
        " which can adversely affect performance. If your model indeed never "
        "has any unused parameters in the forward pass, consider turning this "
        "flag off. Note that this warning may be a false positive if your model "
        "has flow control causing later iterations to have unused parameters.");
  }
```

However, I have seen some users who run large-scale training and have a strong motivation to accelerate trainings are still blind to some useful performance tips like this, mainly because this warning is buried in the noisy log.

### II. Auto-sync Performance Tips and New Features

As we have been adding new features, it will be great to create a portal that can sync relevant performance tips.

For example, when a recent new feature, a DDP communication hook is applied for gradient compression, it may be also beneficial to increase the DDP bucket size to rebalance communication-computation ratio. Or when the network graph is static, we can remind the users to apply delayed AllReduce, which is a new feature released soon.

### III. Exporting Structural Performance Tips to ML Workflow Orchestration Layer

There are many ML workflow orchestration products that already cover PyTorch DDP and probably a few other training libraries/paradigms. If we can collect all the performance tips in a structural way (for DDP as a starting point), both private and OSS orchestration products can use this information to better remind users in certain ways (e.g., email, notification reminder, creating a task for improvement). These enforcement behaviors are more specific to the deployment environment. It will be nice if we can provide a common interface of a set of structural performance tips, which can be fed to the orchestration layer for deploying custom reminders. Additionally, the adoption of these reminders can give a way of tracking performance wins (and probably help quantify the impact of different features and prioritize feature enhancements/development).

The idea is very similar to democratizing a programming style guide (which can certainly include some performance tips). A guidance or some pure documentation won't be very effective, unless each rule can be incorporated into an automation process (e.g., a linter) that can enforce users to apply the suggestions.

## Non-Goals

This proposal aims to complement auto-tuning different configs (at the runtime) rather than replacing it. For some configs, if we can auto-tune them, we should do it rather than requiring any action from the user end. However, not everything can be tuned at the runtime w/o incurring much overheads:
1) Some configs such as NCCL environment variables must be set before DDP is constructed, and hence they are unlikely to be tuned within DDP.
2) It can be prohibitively expensive to tune some configs at the runtime, such as communication hooks.

This proposal does not stop us from tuning some configs that requires more resources at the background, and provide a set of better configs in the reminder. Such tuning can be built as a centralized service in enterprise environment, or we can provide some ways for users to run tuning at the client side. In other words, **our performance tips may not only give some general directions, but can also provide a set of better config values (after offline tuning) or some relevant tuning commands for user to run.**

## Pitch

Define a common interface (essentially a **performance tip container** of DDP) to support registering, displaying, and exporting structural performance tips. Some tips can be developed based on some internal structures of DDP, some of them may require enabling torch profiler.

1. Whenever a performance related warning is triggered in DDP, should also register it in the container, so all the tips can be collected in the same place, not scattered in the log.
2. Every performence tip object should define a `str()` for displaying. Therefore, all the performance tips can be displayed in a batch. The user can call the display API if needed, and by default the display API will be called by DDP destructor.
3. All the performance tips can be exported (to the ML workflow orchestration layer) as a batch. The export method can be customized.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23