Repository: pytorch
Issue: Every process takes a tiny memory on each nvidia GPU card when using DDP.
Labels: oncall: distributed

## 🐛 Bug
Every process of DDP would allocate a tiny memory, it shows 0MiB, on each GPU when use V100 nvidia card. But it is normal on K80 machine. Is that expected? 

## To Reproduce

Steps to reproduce the behavior:
1. run the demo code on v100 * 8 machine.
2. run the demo code on K80 * 4 machine.

### demo code

```python
import torch
import time
import torchvision
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP


def test(local_rank, node_rank, local_size, world_size):
    rank = local_rank + node_rank * local_size
    torch.cuda.set_device(local_rank)
    dist.init_process_group("nccl",
                            init_method="tcp://127.0.0.1:21827",
                            rank=rank,
                            world_size=world_size)
    net = torchvision.models.resnet18()

    net = net.to(local_rank)
    DDP(net, device_ids=[local_rank], output_device=local_rank)
    if rank == 0:
        print("net loaded!")
    time.sleep(100)


def main():
    local_size = torch.cuda.device_count()
    print("local_size: %s" % local_size)
    mp.spawn(test,
             args=(0, local_size, 1 * local_size),
             nprocs=local_size,
             join=True)


if __name__ == '__main__':
    main()
```

###  v100  nvidia-smi shows:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.05    Driver Version: 455.23.05    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:2D:00.0 Off |                    0 |
| N/A   36C    P0    56W / 300W |   1736MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:32:00.0 Off |                    0 |
| N/A   36C    P0    57W / 300W |   1720MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:5B:00.0 Off |                    0 |
| N/A   38C    P0    56W / 300W |   1720MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:5F:00.0 Off |                    0 |
| N/A   35C    P0    56W / 300W |   1740MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  Off  | 00000000:B5:00.0 Off |                    0 |
| N/A   38C    P0    59W / 300W |   1740MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  Off  | 00000000:BE:00.0 Off |                    0 |
| N/A   35C    P0    57W / 300W |   1760MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  Off  | 00000000:E1:00.0 Off |                    0 |
| N/A   41C    P0    59W / 300W |   1760MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  Off  | 00000000:E9:00.0 Off |                    0 |
| N/A   38C    P0    58W / 300W |   1760MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python     1733MiB |
|    0   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    0   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python     1717MiB |
|    1   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    1   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python     1717MiB |
|    2   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    2   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python     1737MiB |
|    3   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    3   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python     1737MiB |
|    4   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    4   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python     1757MiB |
|    5   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    5   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    6   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python     1757MiB |
|    6   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331903      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331904      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331905      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331906      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331907      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331908      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331909      C   ...py3.6-torch1.8/bin/python        0MiB |
|    7   N/A  N/A    331910      C   ...py3.6-torch1.8/bin/python     1757MiB |
+-----------------------------------------------------------------------------+

###  k80  nvidia-smi shows:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:06:00.0 Off |                    0 |
| N/A   66C    P0    62W / 149W |    462MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:07:00.0 Off |                    0 |
| N/A   51C    P0    71W / 149W |    471MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           On   | 00000000:85:00.0 Off |                    0 |
| N/A   65C    P0    58W / 149W |    462MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           On   | 00000000:86:00.0 Off |                    0 |
| N/A   48C    P0    71W / 149W |    471MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     48960      C   ...conda3/envs/py3.6-torch/bin/python   451MiB |
|    1     48961      C   ...conda3/envs/py3.6-torch/bin/python   460MiB |
|    2     48962      C   ...conda3/envs/py3.6-torch/bin/python   451MiB |
|    3     48963      C   ...conda3/envs/py3.6-torch/bin/python   460MiB |
+-----------------------------------------------------------------------------+

## Environment
V100:
 - PyTorch Version (e.g., 1.0): 1.8.1
 - OS (e.g., Linux):  3.10.0-514.44.5.10.h254.x86_64
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Python version: Python 3.7.4
 - CUDA version: cuda11.1
 - Driver Version: 455.23.05
 
K80:
 - PyTorch Version (e.g., 1.0): 1.8.1
 - OS (e.g., Linux):  3.10.0-327.el7.x86_64
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Python version: Python 3.6.2 
 - CUDA version: cuda11.1
 - Driver Version: 418.67



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23