Repository: pytorch
Issue: c10::CUDAError
Labels: oncall: distributed,module: cuda,triaged,module: nccl

## 🐛 Bug
```
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 439, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:  64%|██████████████▊        | 18/28 [00:08<00:04,  2.20it/[W CUDAGuardImpl.h:112] Warning: CUDA warning: the launch timed out and was terminated (function destroyEvent)
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:1055 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f6ea2ee7a22 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so
frame #1: <unknown function> + 0x10983 (0x7f6ea3148983 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a7 (0x7f6ea314a027 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cu.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f6ea2ed15a4 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x2f9 (0x7f6ef9d74ed9 in /opt/conda/lib/python3.site-packages/torch/lib/libtorch_python.so)
frame #5: c10d::Reducer::~Reducer() + 0x276 (0x7f6ef9d6b906 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #6: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7f6ef9d9aa22 in /opt/conda/lib/python3.8/site-pkages/torch/lib/libtorch_python.so)
frame #7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f6ef950ace6 in /opt/conda/lib/python3.8/site-packages/torch/l/libtorch_python.so)
frame #8: std::_Sp_counted_ptr<c10d::Logger*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x1d (0x7f6ef9d9f06d in /opt/conda/lib/python3.8/site-paages/torch/lib/libtorch_python.so)
frame #9: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f6ef950ace6 in /opt/conda/lib/python3.8/site-packages/torch/l/libtorch_python.so)
frame #10: <unknown function> + 0xd88ddf (0x7f6ef9d9cddf in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x4ff4c0 (0x7f6ef95134c0 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #12: <unknown function> + 0x50072e (0x7f6ef951472e in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #13: <unknown function> + 0x12b785 (0x55af59fff785 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1ca984 (0x55af5a09e984 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x11f906 (0x55af59ff3906 in /opt/conda/bin/python)
frame #16: <unknown function> + 0x12bc96 (0x55af59fffc96 in /opt/conda/bin/python)
frame #17: <unknown function> + 0x12bc4c (0x55af59fffc4c in /opt/conda/bin/python)
frame #18: <unknown function> + 0x12bc4c (0x55af59fffc4c in /opt/conda/bin/python)
frame #19: <unknown function> + 0x154ec8 (0x55af5a028ec8 in /opt/conda/bin/python)
frame #20: PyDict_SetItemString + 0x87 (0x55af5a02a127 in /opt/conda/bin/python)
frame #21: PyImport_Cleanup + 0x9a (0x55af5a12a5aa in /opt/conda/bin/python)
frame #22: Py_FinalizeEx + 0x7d (0x55af5a12a94d in /opt/conda/bin/python)
frame #23: Py_RunMain + 0x110 (0x55af5a12b7f0 in /opt/conda/bin/python)
frame #24: Py_BytesMain + 0x39 (0x55af5a12b979 in /opt/conda/bin/python)
frame #25: __libc_start_main + 0xf3 (0x7f6f00d860b3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #26: <unknown function> + 0x1e7185 (0x55af5a0bb185 in /opt/conda/bin/python)

               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95:  79%|██████████████████     | 22/28 [00:10<00:02,  2.44it/ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 3 (pid: 9017) of binary: /opt/conda/bin/python
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
    elastic_launch(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
************************************************
                train.py FAILED
================================================
Root Cause:
[0]:
  time: 2021-11-07_06:11:40
  rank: 3 (local_rank: 3)
  exitcode: -6 (pid: 9017)
  error_file: <N/A>
  msg: "Signal 6 (SIGABRT) received by PID 9017"
================================================
Other Failures:
  <NO_OTHER_FAILURES>
************************************************
```


## To Reproduce

You have to use the two GPU
CUDA:0 (GeForce GTX 1080 Ti, 11178.5MB)
CUDA:1 (GeForce GTX TITAN X, 12212.8125MB)

Steps to reproduce the behavior:
docker pull ultralytics/yolov5
Then using the docker
$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ python -m torch.distributed.launch --nproc_per_node 2 train.py --batch 64 --data coco.yaml --weights yolov5s.pt --device 0,1


## Expected behavior
Train

## Environment
https://github.com/ultralytics/yolov5/blob/master/Dockerfile


## Additional context

<!-- Add any other context about the problem here. -->


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @ngimel