Repository: pytorch
Issue: PyTorch 1.12 cu113 Illegal Memory Access or Internal Error instead of Out of Memory cases
Labels: module: cudnn,module: cuda,triaged

### 🐛 Describe the bug

I'm currently doing benchmark runs on latest timm release with PyTorch 1.12 cu113 (conda) to update timing spreadsheets. I've run into a number of cases where out of memory situations are not being picked up correctly (for illegal memory access, unrecoverable within same process). @ptrblck has mentioned these should be reported. Previous 1.11 cu113 runs were completely clean for contiguous and had only a few channels_last cases.

The illegal memory access are consistent every run (if using same starting batch size), verified across more than one machine.

`CUDA error: an illegal memory access was encountered`:
* vgg11
* vgg13
* vgg16
* vgg19
* nf_regnet_b5

The CUDNN_STATUS_INTERNAL_ERROR model failures below are not as consistent, I have reproduced most of them, but sometimes fail with correct OOM error on a different machine or across same machine restart.

`cuDNN error: CUDNN_STATUS_INTERNAL_ERROR`
* resnetv2_50x3_bitm
* resnetv2_101x3_bitm
* resnetv2_152x2_bit_teacher_384
* resnetv2_152x2_bitm

All failures were recorded on RTX 3090, running `timm` benchmark script (in root of, https://github.com/rwightman/pytorch-image-models) as per example below (with a starting batch size of 1024). Starting w/ a lower batch size for all failure cases will succeed.

ex: `python benchmark.py --amp --model nf_regnet_b5 --bench inference -b 1024`


### Versions

```
Collecting environment information...
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-99-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.4.152
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 470.103.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.22.3          py310hfa59a62_0  
[conda] numpy-base                1.22.3          py310h9585f30_0  
[conda] pytorch                   1.12.0          py3.10_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.13.0              py310_cu113    pytorch
```

cc @csarofeen @ptrblck @xwang233 @ngimel