Repository: pytorch
Issue: DistributedDataParallel high peak memory usage with find_unused_parameters=True
Labels: high priority,triage review,oncall: distributed,triaged,module: ddp

### 🐛 Describe the bug

DDP (via _DDPSink) clones all Tensors in module outputs when find_unused_parameters=True. This can dramatically increase peak memory usage of DDP, resulting in OOM errors for models that previously executed in 1.10.2.

Reproduce with below code under 1.10.2 and1.11.0.
```
import os

import torch
import torch.nn as nn
from torch import distributed as dist
from torch.nn import Module
from torch.nn.parallel import DistributedDataParallel


class Model(Module):
    def __init__(self):
        super().__init__()
        self.conv0 = nn.Conv3d(32, 32, 5)
        self.conv1 = nn.Conv3d(32, 32, 5)
        self.conv2 = nn.Conv3d(32, 32, 5)

    def forward(self, x):
        results = {}
        results[0] = x = self.conv0(x)
        results[1] = x = self.conv1(x)
        results[2] = x = self.conv2(x)
        return x, results


if __name__ == '__main__':
    os.environ['CUDA_VISIBLE_DEVICES'] = str(7)
    os.environ.update(
        MASTER_ADDR='127.0.0.1',
        MASTER_PORT=str(12345),
        WORLD_SIZE=str(1),
        RANK=str(0),
    )
    dist.init_process_group('nccl')
    device = torch.device('cuda')

    for find_unused_parameters in [False, True]:
        model = Model().to(device)
        ddp = DistributedDataParallel(
            model,
            find_unused_parameters=find_unused_parameters,
            gradient_as_bucket_view=True
        )

        input = torch.zeros(8, 32, 129, 129, 129, device=device)

        torch.cuda.reset_peak_memory_stats()

        output, extras = ddp(input)
        output.sum().backward()

        print(f'version={torch.__version__}')
        print(f'find_unused_parameters={find_unused_parameters}')
        print(torch.cuda.memory_summary())

```

Results
```
version=1.10.2
find_unused_parameters=False
...
|---------------------------------------------------------------------------|
| Allocated memory      |    7301 MB |   14636 MB |   36270 MB |   28968 MB |
|       from large pool |    7299 MB |   14635 MB |   36267 MB |   28967 MB |
|       from small pool |       1 MB |       1 MB |       2 MB |       1 MB |
|---------------------------------------------------------------------------|
| Active memory         |    7301 MB |   14636 MB |   36270 MB |   28968 MB |
|       from large pool |    7299 MB |   14635 MB |   36267 MB |   28967 MB |
|       from small pool |       1 MB |       1 MB |       2 MB |       1 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   22562 MB |   22562 MB |   22562 MB |       0 B  |
|       from large pool |   22560 MB |   22560 MB |   22560 MB |       0 B  |
|       from small pool |       2 MB |       2 MB |       2 MB |       0 B  |
...
find_unused_parameters=True
...
|---------------------------------------------------------------------------|
| Allocated memory      |    7300 MB |   14636 MB |   72540 MB |   65240 MB |
|       from large pool |    7299 MB |   14635 MB |   72535 MB |   65235 MB |
|       from small pool |       1 MB |       2 MB |       5 MB |       4 MB |
|---------------------------------------------------------------------------|
| Active memory         |    7300 MB |   14636 MB |   72540 MB |   65240 MB |
|       from large pool |    7299 MB |   14635 MB |   72535 MB |   65235 MB |
|       from small pool |       1 MB |       2 MB |       5 MB |       4 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   22564 MB |   22564 MB |   22564 MB |       0 B  |
|       from large pool |   22560 MB |   22560 MB |   22560 MB |       0 B  |
|       from small pool |       4 MB |       4 MB |       4 MB |       0 B  |
|---------------------------------------------------------------------------|
```
```
version=1.11.0
find_unused_parameters=False
...
|---------------------------------------------------------------------------|
| Allocated memory      |    7301 MB |   14636 MB |   36270 MB |   28968 MB |
|       from large pool |    7299 MB |   14635 MB |   36267 MB |   28967 MB |
|       from small pool |       1 MB |       1 MB |       2 MB |       1 MB |
|---------------------------------------------------------------------------|
| Active memory         |    7301 MB |   14636 MB |   36270 MB |   28968 MB |
|       from large pool |    7299 MB |   14635 MB |   36267 MB |   28967 MB |
|       from small pool |       1 MB |       1 MB |       2 MB |       1 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   22562 MB |   22562 MB |   22562 MB |       0 B  |
|       from large pool |   22560 MB |   22560 MB |   22560 MB |       0 B  |
|       from small pool |       2 MB |       2 MB |       2 MB |       0 B  |
|---------------------------------------------------------------------------|
...
find_unused_parameters=True
...
|---------------------------------------------------------------------------|
| Allocated memory      |    8865 MB |   19271 MB |   79306 MB |   70441 MB |
|       from large pool |    8863 MB |   19268 MB |   79300 MB |   70437 MB |
|       from small pool |       1 MB |       2 MB |       5 MB |       4 MB |
|---------------------------------------------------------------------------|
| Active memory         |    8865 MB |   19271 MB |   79306 MB |   70441 MB |
|       from large pool |    8863 MB |   19268 MB |   79300 MB |   70437 MB |
|       from small pool |       1 MB |       2 MB |       5 MB |       4 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   33628 MB |   33628 MB |   33628 MB |       0 B  |
|       from large pool |   33624 MB |   33624 MB |   33624 MB |       0 B  |
|       from small pool |       4 MB |       4 MB |       4 MB |       0 B  |
|---------------------------------------------------------------------------|
```

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration:
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB
GPU 4: A100-SXM4-40GB
GPU 5: A100-SXM4-40GB
GPU 6: A100-SXM4-40GB
GPU 7: A100-SXM4-40GB

Nvidia driver version: 460.73.01
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.0.5
/usr/lib64/libcudnn_adv_infer.so.8.0.5
/usr/lib64/libcudnn_adv_train.so.8.0.5
/usr/lib64/libcudnn_cnn_infer.so.8.0.5
/usr/lib64/libcudnn_cnn_train.so.8.0.5
/usr/lib64/libcudnn_ops_infer.so.8.0.5
/usr/lib64/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] pytorch-ignite==0.4.8
[pip3] torch==1.11.0
[conda] blas                      2.113                       mkl    conda-forge
[conda] blas-devel                3.9.0            13_linux64_mkl    conda-forge
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] ignite                    0.4.8                      py_0    pytorch
[conda] libblas                   3.9.0            13_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            13_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            13_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            13_linux64_mkl    conda-forge
[conda] mkl                       2022.0.1           h8d4b97c_803    conda-forge
[conda] mkl-devel                 2022.0.1           ha770c72_804    conda-forge
[conda] mkl-include               2022.0.1           h8d4b97c_803    conda-forge
[conda] numexpr                   2.8.0           mkl_py39hba4d566_1    conda-forge
[conda] numpy                     1.22.2           py39h91f2184_0    conda-forge
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang