Repository: pytorch
Issue: Illegal memory access / CUDNN_STATUS_MAPPING_ERROR on Quadro 8000
Labels: module: cudnn,triaged

## 🐛 Bug

When training the model from https://github.com/rosinality/stylegan2-pytorch on a Quadro RTX 8000, in Pytorch 1.4, cuda 10.1, cudnn 7.6.5 I get the error:
`RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR`

This only happens for very large image sizes and large batch sizes, and appears to be exactly the error from [#22496 ](https://github.com/pytorch/pytorch/issues/22496).

To fix it, I tried installing pytorch from master, currently 1.6.0a0+e311e53, along with cuda 10.2, cudnn 7.6.5. Now I get an error under the same circumstances (i.e. only for large image and batch sizes), but the error is different:

```
File "train.py", line 232, in train
    path[0].backward()
  File "/home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/autograd/__init__.py", line 98, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA error: an illegal memory access was encountered (launch_unrolled_kernel at /home/zack/setup/pytorch/aten/src/ATen/native/cuda/CUDALoops.cuh:242)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6c (0x7f8ffcb652cc in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: void at::native::gpu_kernel_impl<__nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 4u>, float (float)> >(at::TensorIterator&, __nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 4u>, float (float)> const&) + 0x1a94 (0x7f8f182b5764 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: void at::native::gpu_kernel<__nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 4u>, float (float)> >(at::TensorIterator&, __nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 4u>, float (float)> const&) + 0x15b (0x7f8f182b6e4b in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: at::native::copy_device_to_device(at::TensorIterator&, bool) + 0x675 (0x7f8f18294eb5 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x1e4800c (0x7f8f1829800c in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x98d99b (0x7f8f29f8199b in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x98a913 (0x7f8f29f7e913 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: at::native::copy_(at::Tensor&, at::Tensor const&, bool) + 0x54 (0x7f8f29f80694 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #8: at::native::contiguous(at::Tensor const&, c10::MemoryFormat) + 0x4a0 (0x7f8f2a252600 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0xf86e88 (0x7f8f2a57ae88 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0xff1d13 (0x7f8f2a5e5d13 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x137faa3 (0x7f8f177cfaa3 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #12: <unknown function> + 0x3163957 (0x7f8f195b3957 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #13: at::native::cudnn_convolution_transpose_backward_input(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xbb (0x7f8f195b3e7b in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #14: <unknown function> + 0x31d3b7e (0x7f8f19623b7e in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #15: <unknown function> + 0x322be1e (0x7f8f1967be1e in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #16: at::native::cudnn_convolution_transpose_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x50f (0x7f8f195b6ecf in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #17: <unknown function> + 0x31d37a5 (0x7f8f196237a5 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #18: <unknown function> + 0x322c0a0 (0x7f8f1967c0a0 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #19: <unknown function> + 0x2b44c5a (0x7f8f2c138c5a in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #20: <unknown function> + 0x2ba21b0 (0x7f8f2c1961b0 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::generated::CudnnConvolutionTransposeBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x41e (0x7f8f2bdcfdce in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #22: <unknown function> + 0x2bc4f5e (0x7f8f2c1b8f5e in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #23: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x1748 (0x7f8f2c1b3d88 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #24: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x4df (0x7f8f2c1b4c1f in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #25: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0xc0 (0x7f8f2c1aa200 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #26: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x50 (0x7f8ff7b4b440 in /home/zack/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #27: <unknown function> + 0xc819d (0x7f8fff2d319d in /home/zack/anaconda3/envs/pt16/bin/../lib/libstdc++.so.6)
frame #28: <unknown function> + 0x76db (0x7f900410f6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #29: clone + 0x3f (0x7f9003e3888f in /lib/x86_64-linux-gnu/libc.so.6)
```

While the code compile custom ops, I've confirmed that if I replace the custom ops with the native versions the error is still the same (though the frames are a little different). The error in both cases happens when calling backwards on a loss which includes a gradient term (i.e. pytorch is taking a second derivative). The exact location is here: https://github.com/rosinality/stylegan2-pytorch/blob/ab37d5da06bc80257b87a0725628cb9f9f8dee18/train.py#L152. If I remove all of these loss terms, the error goes away, so I suspect the error lies in a combination of the architecture of the generator, large tensors, and second derivatives.

So, while this is similar to [#22496 ](https://github.com/pytorch/pytorch/issues/22496) and a few other errors which appear to have been resolved in updated pytorch versions, it seems under these circumstances there are still issues.

## To Reproduce

Steps to reproduce the behavior:

I can make a minimal example if it would be helpful, but is there something obvious I should try first?

## Expected behavior

No errors.

## Environment

```
PyTorch version: 1.6.0a0+e311e53
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.17.0

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000
GPU 4: Quadro RTX 8000
GPU 5: Quadro RTX 8000
GPU 6: Quadro RTX 8000
GPU 7: Quadro RTX 8000

Nvidia driver version: 440.44
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.13.3
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] mkl                       2019.0                   pypi_0    pypi
[conda] mkl-include               2019.0                   pypi_0    pypi
[conda] numpy                     1.18.2                   pypi_0    pypi
[conda] torch                     1.6.0a0+e311e53          pypi_0    pypi
[conda] torchvision               0.6.0a0+dec8628          pypi_0    pypi
```


cc @csarofeen @ptrblck