Repository: pytorch
Issue: Syncbatchnorm and DDP 
Labels: oncall: distributed,module: cuda,triaged

## 🐛 Bug

Running DDP with BatchSyncNorm.
The training will run for a couple of batches and the  all GPUs fall off the bus.
The training runs fine without BatchSyncNorm.

This issue occurs in two models, deeplabv3 and another model, that I have tested so far.

## To Reproduce

```
      os.environ['MASTER_ADDR'] = 'localhost'
      os.environ['MASTER_PORT'] = '123553'
      dist.init_process_group("gloo", rank=rank, world_size=world_size)
      model = models[model](num_classes=ncls)
      model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)
      model = model.cuda(device)
      model = DDP(model, device_ids=[rank], find_unused_parameters=True)
      optimizer = torch.optim.Adam(model.parameters())

      E,B = epochs, len(trn_dtldr)
      for e in range(E):
          for b, batch in enumerate(trn_dtldr):
              x,y = batch['x'].cuda(device), batch['y'].cuda(device)
              py = model(x)

              optimizer.zero_grad()
              l = loss(py,y)
              l.backward()
              optimizer.step()
      cleanup()
```

Nvidia-smi
`Unable to determine the device handle for GPU 0000:17:00.0: GPU is lost.  Reboot the system to recover this GPU`

Nvidia Error Log
```
Aug 26 15:27:32 ailcm-ai1 kernel: [87430.975714] nvidia-uvm: Loaded the UVM driver, major device number 511.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.832707] NVRM: GPU at PCI:0000:17:00: GPU-f114b259-1fd6-a53b-8882-7d59735e0271
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.832733] NVRM: GPU Board Serial Number: 
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.832736] NVRM: Xid (PCI:0000:17:00): 79, pid=24953, GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.850885] NVRM: GPU 0000:17:00.0: GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.850886] NVRM: GPU 0000:17:00.0: GPU is on Board .
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.850896] NVRM: A GPU crash dump has been created. If possible, please run
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.850896] NVRM: nvidia-bug-report.sh as root to collect this data before
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.850896] NVRM: the NVIDIA kernel module is unloaded.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.851013] NVRM: GPU at PCI:0000:65:00: GPU-ec0ca56f-dbab-4d7f-8341-0bfd7d5f8a24
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.851016] NVRM: GPU Board Serial Number: 
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.851019] NVRM: Xid (PCI:0000:65:00): 79, pid=24955, GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.851117] NVRM: GPU 0000:65:00.0: GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.851119] NVRM: GPU 0000:65:00.0: GPU is on Board .
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852082] NVRM: GPU at PCI:0000:18:00: GPU-499e9a48-65fb-b44f-21ef-1475bafeba1e
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852083] NVRM: GPU Board Serial Number: 
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852085] NVRM: Xid (PCI:0000:18:00): 79, pid=24955, GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852179] NVRM: GPU 0000:18:00.0: GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852181] NVRM: GPU 0000:18:00.0: GPU is on Board .
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852187] NVRM: GPU at PCI:0000:b4:00: GPU-de26af30-1122-e55a-1bb5-ed7732106434
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852189] NVRM: GPU Board Serial Number: 
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852191] NVRM: Xid (PCI:0000:b4:00): 79, pid=24955, GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852286] NVRM: GPU 0000:b4:00.0: GPU has fallen off the bus.
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.852287] NVRM: GPU 0000:b4:00.0: GPU is on Board .
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.937436] nvidia-gpu 0000:b4:00.3: Refused to change power state, currently in D3
Aug 26 15:28:16 ailcm-ai1 kernel: [87474.997587] nvidia-gpu 0000:18:00.3: Refused to change power state, currently in D3
Aug 26 15:28:16 ailcm-ai1 kernel: [87475.057545] nvidia-gpu 0000:65:00.3: Refused to change power state, currently in D3
Aug 26 15:28:17 ailcm-ai1 kernel: [87476.298279] nvidia-gpu 0000:18:00.3: i2c timeout error ffffffff
Aug 26 15:28:17 ailcm-ai1 kernel: [87476.298427] nvidia-gpu 0000:b4:00.3: i2c timeout error ffffffff
Aug 26 15:28:17 ailcm-ai1 kernel: [87476.298570] nvidia-gpu 0000:65:00.3: i2c timeout error ffffffff
```

Python ErrorLog
```
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: unspecified launch failure
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f8a406c51e2 in /home/yhong/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f8a40913f92 in /home/yhong/.local/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f8a406b39cd in /home/yhong/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x540ae2 (0x7f8a8c24bae2 in /home/yhong/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x540b86 (0x7f8a8c24bb86 in /home/yhong/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: /usr/bin/python3() [0x54f226]
frame #6: /usr/bin/python3() [0x5885c5]
frame #7: /usr/bin/python3() [0x588a98]
frame #8: /usr/bin/python3() [0x5ad558]
frame #9: /usr/bin/python3() [0x5ad56e]
frame #10: /usr/bin/python3() [0x5ad56e]
frame #11: /usr/bin/python3() [0x5ad56e]
frame #12: /usr/bin/python3() [0x5ad56e]
frame #13: /usr/bin/python3() [0x5ad56e]
frame #14: /usr/bin/python3() [0x5ad56e]
frame #15: /usr/bin/python3() [0x5ad56e]
frame #16: /usr/bin/python3() [0x5ad56e]
frame #17: /usr/bin/python3() [0x5ad56e]
frame #18: /usr/bin/python3() [0x5ad56e]
frame #19: /usr/bin/python3() [0x58fe99]
frame #20: /usr/bin/python3() [0x58fe36]
frame #21: /usr/bin/python3() [0x6290a3]
frame #22: _PyEval_EvalFrameDefault + 0x4488 (0x510238 in /usr/bin/python3)
frame #23: /usr/bin/python3() [0x509918]
frame #24: /usr/bin/python3() [0x50a64d]
frame #25: _PyEval_EvalFrameDefault + 0x444 (0x50c1f4 in /usr/bin/python3)
frame #26: /usr/bin/python3() [0x509918]
frame #27: /usr/bin/python3() [0x50a64d]
frame #28: _PyEval_EvalFrameDefault + 0x444 (0x50c1f4 in /usr/bin/python3)
frame #29: /usr/bin/python3() [0x507f24]
frame #30: /usr/bin/python3() [0x509c50]
frame #31: /usr/bin/python3() [0x50a64d]
frame #32: _PyEval_EvalFrameDefault + 0x1226 (0x50cfd6 in /usr/bin/python3)
frame #33: /usr/bin/python3() [0x507f24]
frame #34: PyRun_StringFlags + 0xaf (0x63501f in /usr/bin/python3)
frame #35: PyRun_SimpleStringFlags + 0x3d (0x63848d in /usr/bin/python3)
frame #36: Py_Main + 0x353 (0x638fa3 in /usr/bin/python3)
frame #37: main + 0xe0 (0x4b0dc0 in /usr/bin/python3)
frame #38: __libc_start_main + 0xe7 (0x7f8a8fc6bb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #39: _start + 0x2a (0x5b26fa in /usr/bin/python3)

```

## Expected behavior

Expected for training to work well.

## Environment

```

Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] Could not collect

```

 - How you installed PyTorch (`conda`, `pip`, source): pip
 - CUDA/cuDNN version:
 - GPU models and configuration: 4 gpu 2080ti with 1700 power supply and 100+gb ram

## Additional context



cc @ngimel @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski