Repository: pytorch
Issue: Distributed tests fail with pytest
Labels: oncall: distributed,triaged

## 🐛 Bug

This might be similar or the same as #19177. I'm getting test failures in the distributed tests. The failing tests are:
```
DistributedDataParallelTest.test_accumulate_gradients_module
DistributedDataParallelTest.test_accumulate_gradients_no_sync
DistributedDataParallelTest.test_arbitrary_forward_return_value
DistributedDataParallelTest.test_ddp_multi_device_module_config
DistributedDataParallelTest.test_failure_recovery
DistributedDataParallelTest.test_find_unused_parameters_kwarg
DistributedDataParallelTest.test_fp16
DistributedDataParallelTest.test_global_local_unused_params_grad
DistributedDataParallelTest.test_gloo_backend_1gpu_module_device_ids_integer_list
DistributedDataParallelTest.test_gloo_backend_1gpu_module_device_ids_torch_device_list
DistributedDataParallelTest.test_gloo_backend_2gpu_module
DistributedDataParallelTest.test_gloo_backend_cpu_module
DistributedDataParallelTest.test_grad_layout_1devicemodule_1replicaperprocess
DistributedDataParallelTest.test_grad_layout_2devicemodule
DistributedDataParallelTest.test_ignored_output
DistributedDataParallelTest.test_ignored_output_with_unused_parameters
DistributedDataParallelTest.test_multiple_outputs_multiple_backward
DistributedDataParallelTest.test_nccl_backend_1gpu_module_device_ids_integer_list
DistributedDataParallelTest.test_nccl_backend_1gpu_module_device_ids_torch_device_list
DistributedDataParallelTest.test_nccl_backend_2gpu_module
DistributedDataParallelTest.test_no_grad
DistributedDataParallelTest.test_param_layout_mismatch_error
DistributedDataParallelTest.test_sparse_gradients
CommTest.test_broadcast_coalesced_gloo_cuda
CommTest.test_broadcast_coalesced_nccl
```

## To Reproduce

Steps to reproduce the behavior:

1. `cd test && python run_test.py --verbose --pytest -x test_cpp_extensions_aot_no_ninja test_cpp_extensions_aot_ninja test_cpp_extensions_jit`

Using pytest to avoid spurious failures in other tests which go away using pytest. The excluded ones fail on POWER9.

The logs are pretty much mangled (likely due to parallel processes writing to stdout), but what I could see are `RuntimeError: CUDA error: initialization error` messages.

## Environment

PyTorch version: 1.6.0-rc2
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.6 (Maipo)
GCC version: (GCC) 8.3.0
CMake version: version 3.15.3

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB

Nvidia driver version: 440.64.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.17.3
[conda] Could not collect


## Additional context

This is on a POWER9 node. Currently testing on an x86 node using the same software versions.

Edit: Getting the same errors on an x86 node with K80s


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski