Repository: pytorch
Issue: 8 tests in test_c10d fail when running all tests in one command
Labels: oncall: distributed,module: tests,triaged

## 🐛 Bug

All tests in test_c10d pass if run individually in separate commands, but 8 fail consistently if run together in one pytest command. They all hit the similar error as shown below:

```
test/test_c10d.py::DistributedDataParallelTest::test_sync_reduction Process process 0:                                                                                 
Traceback (most recent call last):                                                                                                                                     
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap                                                                
    self.run()                                                                                                                                                         
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run                                                                        
    self._target(*self._args, **self._kwargs)                                               
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 481, in _run
    getattr(self, self.id().split(".")[2])() 
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 439, in wrapper
    fn(self)                                   
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 66, in wrapper
    return func(*args, **kwargs)                       
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 55, in wrapper                                                                                             
    return func(*args, **kwargs)                                                                                                                                       
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 1802, in test_sync_reduction                                                                              
    for d in devices]                     
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 1802, in <listcomp>
    for d in devices]                                                                                                                                                  
RuntimeError: CUDA error: initialization error           
Process process 1:                                                  
Traceback (most recent call last):                               
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap                                                                
    self.run()                                                                                                                                                         
  File "/home/shenli/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)                                                                                                                         
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 481, in _run
    getattr(self, self.id().split(".")[2])()                     
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 439, in wrapper
    fn(self)
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 66, in wrapper
    return func(*args, **kwargs)
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 1802, in test_sync_reduction
    for d in devices]
  File "/data/users/shenli/pytorch/test/test_c10d.py", line 1802, in <listcomp>
    for d in devices]
RuntimeError: CUDA error: initialization error
```

The failed ones are:

```
1. DistributedDataParallelTest.test_dist_broadcast_coalesced_gloo
2. DistributedDataParallelTest.test_fp16
3. DistributedDataParallelTest.test_gloo_backend
4. DistributedDataParallelTest.test_nccl_backend
5. DistributedDataParallelTest.test_queue_reduction
6. DistributedDataParallelTest.test_sync_params_no_buffers
7. DistributedDataParallelTest.test_sync_params_with_buffers
8. DistributedDataParallelTest.test_sync_reduction
```

## To Reproduce

`py.test test/test_c10d.py `

## Environment

Collecting environment information...
PyTorch version: 1.1.0a0+91a2900
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: CentOS Linux 7 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 396.69
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.16.2
[pip] torch==1.1.0a0+91a2900
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.3                      199  
[conda] mkl-include               2019.3                      199  
[conda] mkl_fft                   1.0.10           py37ha843d7b_0  
[conda] mkl_random                1.0.2            py37hd81dba3_0  
[conda] torch                     1.1.0a0+91a2900           dev_0    <develop>