Repository: pytorch
Issue: DistributedDataParallel creates too many threads
Labels: oncall: distributed,module: dataloader,better-engineering,module: ddp

### 🐛 Describe the bug

Hi! I ran into an issue that after running `torch.nn.parallel.DistributedDataParallel` the process creates too many threads (so it’s not possible to run training with num_worker>0 - “can't start new thread” error)

## To Reproduce

```python
import os
import torch.distributed as dist
import torch.utils.data.distributed
import time
import torchvision
import torchvision.transforms as transforms


if __name__ == '__main__':
    rank = int(os.environ["SLURM_PROCID"])
    gpu = rank

    dist.init_process_group(backend='nccl', init_method='env://',
                            world_size=8, rank=rank)

    torch.cuda.set_device(gpu)
    model = torch.nn.Conv2d(128, 128, (5, 5))
    model.cuda(gpu)

    time.sleep(30)  # check number of threads here

    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[gpu])

    print('after DistributedDataParallel', flush=True)
    time.sleep(30)  # check number of threads here

    transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    dataset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    dataloader = torch.utils.data.DataLoader(dataset, num_workers=1, batch_size=32)
    for data in dataloader:
        pass
```

```bash
#!/bin/bash

export MASTER_PORT=14340
export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)

export OMP_NUM_THREADS='1'
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL

srun python debug.py
```

### Output: 
```
Traceback (most recent call last):
  File "debug.py", line 29, in <module>
    for data in dataloader:
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 944, in __init__
    self._reset(loader, first_iter=True)
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 975, in _reset
    self._try_put_index()
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1220, in _try_put_index
    self._index_queues[worker_queue_idx].put((self._send_idx, index))
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/multiprocessing/queues.py", line 87, in put
    self._start_thread()
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/multiprocessing/queues.py", line 169, in _start_thread
    self._thread.start()
  File "/scratch/ssml/shvetsova/miniconda3_py37_4.9.2/envs/pytorch_self_superv/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
```

If I check a number of threads per process (`ps -eLf  | grep <PID> | grep -v grep | wc`) before DistributedDataParallel it's **6**, but after DistributedDataParallel it's **74**.




### Versions

PyTorch version: 1.10.0+rocm4.2
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 4.2.21155-37cb3a34

OS: Scientific Linux release 7.9 (Nitrogen) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.45.1.el7.x86_64-x86_64-with-redhat-7.9-Nitrogen
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: Vega 20
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 3.27.5
MIOpen runtime version: 2.11.0

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.10.0+rocm4.2
[pip3] torchvision==0.11.1+rocm4.2
[conda] blas                      1.0                         mkl  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py36he8ac12f_0  
[conda] mkl_fft                   1.3.0            py36h54f3939_0  
[conda] mkl_random                1.1.1            py36h0573a6f_0  
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] numpy-base                1.19.2           py36hfa32c7d_0  
[conda] torch                     1.10.0+rocm4.2           pypi_0    pypi
[conda] torchvision               0.11.1+rocm4.2           pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @SsnL @VitalyFedyunin @ejguan @NivekT