Repository: pytorch
Issue: PyTorch 1.6 DataParallel causes CUDNN_STATUS_BAD_PARAM in backward pass
Labels: module: cudnn,module: cuda,triaged,module: data parallel

## 🐛 Bug

On a machine having 2 different GPUs DataParallel causes a crash in the C++ backed with RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM error message. With disabled CUDNN it works fine. 

The crash is stochastic and the probability depends on the used model. Simpler convnets works fine. The probability is related to the number of batchnorms used in the model, however, I was unable to figure out how exactly (it is not monotonic). An easy way to test this is to start with a convnet without batchnorm, and start stacking batchnoms at an arbitrary position in the network. It will start crashing with a certain probability. After enough batchnorms it crashes with P=1.

## To Reproduce

Running the following script

```python
import torch
from torchvision.models import resnet101

rnn = resnet101(num_classes=10).cuda()
par = torch.nn.DataParallel(rnn)

inp = torch.randn(100, 3, 32, 32).cuda()

par(inp).sum().backward()
```
Causes the following crash:

```
/home/robert/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:26: UserWarning: 
    There is an imbalance between your GPUs. You may want to exclude GPU 1 which
    has less than 75% of the memory or cores of GPU 0. You can do so by setting
    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES
    environment variable.
  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))
Traceback (most recent call last):
  File "crash.py", line 9, in <module>
    par(inp).sum().backward()
  File "/home/robert/.local/lib/python3.8/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/robert/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 125, in backward
    Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM
Exception raised from operator() at /pytorch/aten/src/ATen/native/cudnn/Conv.cpp:1141 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f3d36ba51e2 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xebb822 (0x7f3d37ec9822 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xebd985 (0x7f3d37ecb985 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xebe24e (0x7f3d37ecc24e in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xebe910 (0x7f3d37ecc910 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::cudnn_convolution_backward_weight(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0x49 (0x7f3d37eccb69 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xf1f53b (0x7f3d37f2d53b in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xf4f178 (0x7f3d37f5d178 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #8: at::cudnn_convolution_backward_weight(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0x1ad (0x7f3d72e052ad in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x18a (0x7f3d37ec678a in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xf1f445 (0x7f3d37f2d445 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #11: <unknown function> + 0xf4f1d4 (0x7f3d37f5d1d4 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #12: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7f3d72e14242 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2ec9c62 (0x7f3d74ad7c62 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2ede224 (0x7f3d74aec224 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #15: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7f3d72e14242 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x258 (0x7f3d7495ec38 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x3375bb7 (0x7f3d74f83bb7 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f3d74f7f400 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f3d74f7ffa1 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #20: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f3d74f78119 in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7f3d8271386a in /home/robert/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0xcfb74 (0x7f3d8365cb74 in /usr/lib/libstdc++.so.6)
frame #23: <unknown function> + 0x9422 (0x7f3d84903422 in /usr/lib/libpthread.so.0)
frame #24: clone + 0x43 (0x7f3d84a1bbf3 in /usr/lib/libc.so.6)
```

## Expected behavior

Don't crash. If for some reason CUDNN requires to have identical GPUs if running in parallel, then display a runtime error that is easier to interpret than this. I assumed the error is in my code and lost a few hours debugging it.

## Environment

Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2

OS: Manjaro Linux (x86_64)
GCC version: (GCC) 10.1.0
Clang version: Could not collect
CMake version: version 3.17.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 440.82
cuDNN version: /usr/lib/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.6.0
[pip3] torchfile==0.1.0
[pip3] torchvision==0.7.0
[conda] Could not collect

## Additional context

I tried reproducing the error on other machines having identical GPUs and I failed. That is not the only difference nor in hardware nor in the software environment, so it is hard to compare.

cc @ngimel @csarofeen @ptrblck @xwang233