Repository: pytorch
Issue: Reproducible "CUDA error: an illegal memory access was encountered"
Labels: needs reproduction,module: crash,module: cuda,triaged

### 🐛 Describe the bug

I get a "CUDA error: an illegal memory access was encountered" error 100% of the time with the following code.

The error seems to occur only when `input_length` is 249 or 250. For other values, it doesn't seem to happen.

I'm guessing this is hitting some edge case in tensor manipulation & indexing, but I have no idea where.

```
#!/usr/bin/env python3

import torch
from torch import nn
from torchaudio.models import Conformer


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        encoder_dim = 272
        num_heads = 8
        num_layers = 18
        conv_kernel_size = 31
        num_labels = 500

        conv_layers = [
            nn.Conv1d(
                in_channels=80, out_channels=encoder_dim * 2,
                kernel_size=3, stride=1, padding=0, dilation=1),
            nn.GLU(dim=1),
            nn.Conv1d(
                in_channels=encoder_dim, out_channels=encoder_dim * 2,
                kernel_size=3, stride=2, padding=0, dilation=1),
            nn.GLU(dim=1),
            nn.Conv1d(
                in_channels=encoder_dim, out_channels=encoder_dim * 2,
                kernel_size=3, stride=1, padding=0, dilation=1),
            nn.GLU(dim=1),
            nn.Conv1d(
                in_channels=encoder_dim, out_channels=encoder_dim * 2,
                kernel_size=3, stride=2, padding=0, dilation=1),
            nn.GLU(dim=1),
        ]

        self.conv_subsampler = nn.Sequential(*conv_layers)

        self.conformer = Conformer(
            input_dim=encoder_dim, num_heads=num_heads, ffn_dim=4 * encoder_dim,
            num_layers=num_layers, depthwise_conv_kernel_size=conv_kernel_size,
        )

        self.output_layer = nn.Linear(encoder_dim, num_labels + 1)

    def forward(self, input_t: torch.Tensor):
        output = self.conv_subsampler(input_t.unsqueeze(1).transpose(2, 3).squeeze(1)).transpose(1, 2)
        output, output_lengths = self.conformer(output, torch.tensor([output.shape[1]]).cuda())
        output = self.output_layer(output)
        return output, output_lengths


# 249 or 250 here causes "RuntimeError: CUDA error: an illegal memory access was encountered"
input_length = 250

input_tensor = torch.rand((1, input_length, 80), dtype=torch.float32).cuda()
model = Model().cuda()
output, output_len = model(input_tensor)
output = nn.functional.log_softmax(output, dim=2).transpose(0, 1)
lossfun = nn.CTCLoss(blank=0, reduction='mean')
print(output.shape)
loss = lossfun(output, torch.cuda.IntTensor([[1, 2]]), output_len, torch.cuda.IntTensor([2]))
loss.backward()
```

Output:
```
$ reproduce_250_bug.py
torch.Size([60, 1, 501])
Traceback (most recent call last):
  File "reproduce_250_bug.py", line 62, in <module>
    loss.backward()
  File "/home/wjeon/miniconda3/envs/py38/lib/python3.8/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/wjeon/miniconda3/envs/py38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: an illegal memory access was encountered
```


### Versions

```
$ python collect_env.py
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-194-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: Tesla V100-SXM2-32GB
Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.2
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.23.2                   pypi_0    pypi
[conda] numpy-base                1.23.1           py38ha15fc14_0
[conda] pytorch                   1.12.1          py3.8_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1               py38_cu113    pytorch
[conda] torchvision               0.13.1               py38_cu113    pytorch
```

cc @ezyang @gchanan @zou3519 @ngimel