Repository: pytorch
Issue: 3 Times memory cost when loading the model to torch.nn.parallel.DistributedDataParallel
Labels: oncall: distributed,module: ddp

Firstly I load the model in to GPU memory, and the information is as follows:
```
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
Sun Aug  1 12:38:30 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:0B:00.0 Off |                    0 |
| N/A   22C    P0    30W / 250W |   3741MiB / 16280MiB |     12%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:81:00.0 Off |                    0 |
| N/A   27C    P0    31W / 250W |   3269MiB / 16280MiB |     21%   E. Process |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     32400      C   ...ghe/anaconda3/envs/leizhikai/bin/python  3731MiB |
|    1     32401      C   ...ghe/anaconda3/envs/leizhikai/bin/python  3275MiB |
+-----------------------------------------------------------------------------+
```

And then after I run the command:
`model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True)`
The memory cost is as follows:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:0B:00.0 Off |                    0 |
| N/A   22C    P0    33W / 250W |   9283MiB / 16280MiB |     92%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:81:00.0 Off |                    0 |
| N/A   27C    P0    34W / 250W |   9221MiB / 16280MiB |    100%   E. Process |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     32400      C   ...ghe/anaconda3/envs/leizhikai/bin/python  9273MiB |
|    1     32401      C   ...ghe/anaconda3/envs/leizhikai/bin/python  9211MiB |
+-----------------------------------------------------------------------------+
```
It costs 3 times memory, I think it's unexpectable.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23