Repository: pytorch
Issue: DistributedDataParallel `static_graph=True` fails to handle unused parameters
Labels: oncall: distributed,triaged,module: ddp

### 🐛 Describe the bug

DistributedDataParallel fails to properly handle unused parameters. 

```python

import os

os.environ['TORCH_DISTRIBUTED_DEBUG'] = 'DETAIL'

import torch
import torch.nn as nn
import torch.distributed as dist
from torch.nn import Module
from torch.nn.parallel.distributed import DistributedDataParallel


class M(Module):
    def __init__(self):
        super().__init__()
        self.a = nn.Conv3d(1, 1, 1)
        self.b = nn.Conv3d(1, 1, 1)

    def forward(self, x):
        return self.a(x), self.b(x)


if __name__ == '__main__':
    os.environ.update(
        MASTER_ADDR='127.0.0.1',
        MASTER_PORT=str(12345),
        WORLD_SIZE=str(1),
        RANK=str(0),
    )
    dist.init_process_group('nccl')

    m = M().cuda()
    ddp = DistributedDataParallel(
        m,
        static_graph=True,
        find_unused_parameters=False  # default
    )

    x = torch.zeros(1, 1, 1, 1, 1, device='cuda')

    for i in range(5):
        print(f'iter: {i}')
        a, b = ddp(x)
        l = a.sum()
        l.backward()

```


Output

```
iter: 0
iter: 1
iter: 2
Traceback (most recent call last):
  File <...>, in <module>
    a, b = ddp(x)
  File "<...>/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "<...>/conda/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 947, in forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your training graph has changed in this iteration, e.g., one parameter is used in first iteration, but then got unused in the second iteration. this is not compatible with static_graph set to True.
Parameters which did not receive grad for rank 0: b.bias, b.weight
Parameter indices which did not receive grad for rank 0: 2 3
```

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)
Clang version: Could not collect
CMake version: version 3.23.1
Libc version: glibc-2.17

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration:
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB
GPU 4: A100-SXM4-40GB
GPU 5: A100-SXM4-40GB
GPU 6: A100-SXM4-40GB
GPU 7: A100-SXM4-40GB

Nvidia driver version: 460.73.01
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.4.0
/usr/lib64/libcudnn_adv_infer.so.8.4.0
/usr/lib64/libcudnn_adv_train.so.8.4.0
/usr/lib64/libcudnn_cnn_infer.so.8.4.0
/usr/lib64/libcudnn_cnn_train.so.8.4.0
/usr/lib64/libcudnn_ops_infer.so.8.4.0
/usr/lib64/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] pytorch-ignite==0.4.9
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] blas                      2.114                       mkl    conda-forge
[conda] blas-devel                3.9.0            14_linux64_mkl    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] ignite                    0.4.9                      py_0    pytorch
[conda] libblas                   3.9.0            14_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            14_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            14_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            14_linux64_mkl    conda-forge
[conda] mkl                       2022.0.1           h06a4308_117
[conda] mkl-devel                 2022.0.1           h66538d2_117
[conda] mkl-include               2022.0.1           h06a4308_117
[conda] numpy                     1.22.3           py39hc58783e_2    conda-forge
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.12.0               py39_cu113    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501