Repository: pytorch
Issue: test/distributed/test_c10d.py::DistributedDataParallelTest only passes if run before other test cases
Labels: oncall: distributed,triaged

## 🐛 Bug

`test/distributed/test_c10d.py::DistributedDataParallelTest` only passes if run before other test cases that (in-)directly call `torch.cuda.init()`.

## To Reproduce

```
python -m unittest \
    test_c10d.RendezvousEnvTest.test_common_errors \
    test_c10d.DistributedDataParallelTest.test_ddp_comm_hook_sparse_gradients
```


<details><summary>Traceback</summary>
<p>

```
.terminate called after throwing an instance of 'c10::Error'
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
Exception raised from getDevice at ../c10/cuda/impl/CUDAGuardImpl.h:37 (most recent call first):
frame #0: <unknown function> + 0x103b37 (0x7ffb457d0b37 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #1: std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>::operator()() const + 0x4d (0x7ffb7682cbdd in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #2: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x41 (0x7ffb457cf867 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #3: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x7e (0x7ffb457cdeb1 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #4: <unknown function> + 0x22bab (0x7ffb45852bab in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x22e19 (0x7ffb45852e19 in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #6: <unknown function> + 0x537c085 (0x7ffb76715085 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4b (0x7ffb767136d1 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x71 (0x7ffb7cf9e1bf in /home/pmeier/pytorch/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x539a6b7 (0x7ffb767336b7 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x539a4ef (0x7ffb767334ef in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x539a33f (0x7ffb7673333f in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x539a097 (0x7ffb76733097 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #13: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (torch::autograd::Engine::*)(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool), torch::autograd::Engine*, int, std::shared_ptr<torch::autograd::ReadyQueue>, bool> > >::_M_run() + 0x1c (0x7ffb76732f28 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0xc9067 (0x7ffb8c443067 in /home/pmeier/.conda/envs/pytorch-cuda-dev/lib/libstdc++.so.6)
frame #15: <unknown function> + 0x76db (0x7ffbb0be66db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #16: clone + 0x3f (0x7ffbaff6271f in /lib/x86_64-linux-gnu/libc.so.6)

  what():  CUDA error: initialization error
Exception raised from getDevice at ../c10/cuda/impl/CUDAGuardImpl.h:37 (most recent call first):
frame #0: <unknown function> + 0x103b37 (0x7ffb457d0b37 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #1: std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>::operator()() const + 0x4d (0x7ffb7682cbdd in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #2: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x41 (0x7ffb457cf867 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #3: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x7e (0x7ffb457cdeb1 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #4: <unknown function> + 0x22bab (0x7ffb45852bab in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x22e19 (0x7ffb45852e19 in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #6: <unknown function> + 0x537c085 (0x7ffb76715085 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4b (0x7ffb767136d1 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x71 (0x7ffb7cf9e1bf in /home/pmeier/pytorch/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x539a6b7 (0x7ffb767336b7 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x539a4ef (0x7ffb767334ef in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x539a33f (0x7ffb7673333f in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x539a097 (0x7ffb76733097 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #13: std::thread::_State_impl<std::thterminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
Exception raised from getDevice at ../c10/cuda/impl/CUDAGuardImpl.h:37 (most recent call first):
frame #0: <unknown function> + 0x103b37 (0x7ffb457d0b37 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #1: std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>::operator()() const + 0x4d (0x7ffb7682cbdd in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #2: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x41 (0x7ffb457cf867 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #3: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x7e (0x7ffb457cdeb1 in /home/pmeier/pytorch/torch/lib/libc10.so)
frame #4: <unknown function> + 0x22bab (0x7ffb45852bab in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x22e19 (0x7ffb45852e19 in /home/pmeier/pytorch/torch/lib/libc10_cuda.so)
frame #6: <unknown function> + 0x537c085 (0x7ffb76715085 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4b (0x7ffb767136d1 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x71 (0x7ffb7cf9e1bf in /home/pmeier/pytorch/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x539a6b7 (0x7ffb767336b7 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x539a4ef (0x7ffb767334ef in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x539a33f (0x7ffb7673333f in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x539a097 (0x7ffb76733097 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #13: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (torch::autograd::Engine::*)(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool), torch::autograd::Engine*, int, std::shared_ptr<torch::autograd::ReadyQueue>, bool> > >::_M_run() + 0x1c (0x7ffb76732f28 in /home/pmeier/pytorch/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0xc9067 (0x7ffb8c443067 in /home/pmeier/.conda/envs/pytorch-cuda-dev/lib/libstdc++.so.6)
frame #15: <unknown function> + 0x76db (0x7ffbb0be66db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #16: clone + 0x3f (0x7ffbaff6271f in /lib/x86_64-linux-gnu/libc.so.6)

F
======================================================================
FAIL: test_ddp_comm_hook_sparse_gradients (test_c10d.DistributedDataParallelTest)
Runs "test_sparse_gradients" unit test with DDP communication hook. We define a
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 286, in wrapper
    self._join_processes(fn)
  File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 418, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 477, in _check_return_codes
    self.assertEqual(
  File "/home/pmeier/pytorch/torch/testing/_internal/common_utils.py", line 1234, in assertEqual
    super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
AssertionError: False is not true : Scalars failed to compare as equal! Comparing -6 and 0 gives a difference of 6, but the allowed difference with rtol=0 and atol=0 is only 0!
Expected zero exit code but got -6
```
</p>
</details>

## Expected behavior

Run without error independent of execution order.

## Environment

```
PyTorch version: 1.9.0a0+git34d9278
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (crosstool-NG 1.24.0.133_b0863d8_dirty) 9.3.0
Clang version: 11.0.1 (https://github.com/conda-forge/clangdev-feedstock 9fbb64b62ff49e9d206a06e62453b27557b3ed73)
CMake version: version 3.19.4

Python version: 3.9 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000

Nvidia driver version: 460.27.04
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-10.2.89/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.0
[pip3] torch==1.9.0a0+git34d9278
[conda] magma-cuda110             2.5.2                         1    pytorch
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-include               2020.4             h726a3e6_304    conda-forge
[conda] numpy                     1.20.0           py39hdbf815f_0    conda-forge
[conda] torch                     1.9.0a0+git34d9278           dev_0    <develop>
```

## Additional context

- This was first reported in #19177 and again in #41337. This was attributed to `pytest`, but this only came up there, because `pytest` uses another execution order than `unittest`. 
- The other issus only report a number of test that fail. This is due to another bug in the test setup. After #53502 is merged every test in `DistributedDataParallelTest` that does not check for a specific error to be raised is failing with the above error.
- When applying the patch from #53502 we can get a better error message by running

    ```
    python -m unittest \
        test_c10d.RendezvousEnvTest.test_common_errors \
        test_c10d.DistributedDataParallelTest.test_cummulate_gradients_module
    ```

    <details><summary>Traceback</summary>
    <p>

    ```
    Process 1 exited with error code 10 and exception:
    Traceback (most recent call last):
      File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 364, in run_test
        getattr(self, test_name)()
      File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 286, in wrapper
        fn()
      File "/home/pmeier/pytorch/torch/testing/_internal/common_distributed.py", line 71, in wrapper
        return func(*args, **kwargs)
      File "/home/pmeier/pytorch/test/distributed/test_c10d.py", line 3165, in test_accumulate_gradients_module
        self._test_accumulate_gradients_module()
      File "/home/pmeier/pytorch/test/distributed/test_c10d.py", line 3118, in _test_accumulate_gradients_module
        model, ddp_model, input, target = self._prepare_single_device_module(
      File "/home/pmeier/pytorch/test/distributed/test_c10d.py", line 2250, in _prepare_single_device_module
        copy.deepcopy(model).to(devices[0]),
      File "/home/pmeier/pytorch/torch/nn/modules/module.py", line 816, in to
        return self._apply(convert)
      File "/home/pmeier/pytorch/torch/nn/modules/module.py", line 530, in _apply
        module._apply(fn)
      File "/home/pmeier/pytorch/torch/nn/modules/module.py", line 552, in _apply
        param_applied = fn(param)
      File "/home/pmeier/pytorch/torch/nn/modules/module.py", line 814, in convert
        return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
      File "/home/pmeier/pytorch/torch/cuda/__init__.py", line 160, in _lazy_init
        raise RuntimeError(
    RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
    ```

    </p>
    </details>

- Following the instructions in the error message, the error goes away if we patch `DistributedDataParallelTest.setup()`

    https://github.com/pytorch/pytorch/blob/34d9278c1913d83db47a25f0cac71b69ab877c84/test/distributed/test_c10d.py#L2220-L2225

    to always use `self._spawn_processes()`.


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu