Repository: pytorch
Issue: RuntimeError: cuDNN version incompatibility: happened when I have lstm layer
Labels: module: build,module: cudnn,triaged

### 🐛 Describe the bug

I have a simple deep learning learning model like this
```
class model(nn.Module):
    def __init__(self, insize=32, hiddensize = 16, level=10):
        super(model, self).__init__()

        self.conv = nn.Conv2d(1, 32, kernel_size=(1,20))
        self.lstm = nn.LSTM(input_size=insize, hidden_size=hiddensize, batch_first=True)
        self.fc1 = nn.Linear(in_features=hiddensize, out_features=3)

    def forward(self, x):
        out = self.conv(x)
        
        out = out.permute(0, 2, 1, 3)
        out = out.squeeze(3)
        out, _ = self.lstm(out)

        out = torch.narrow(out, 1, out.shape[1]-1, 1).squeeze(1)
        
        out = self.fc1(out)

        return out
```
The input I used is a 100*20 image with 1 channel.
When I run the model,  the error happened
```
RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 3, 2) but found runtime version (8, 2, 4). 
PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. 
Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 3, 2)
```
But when I disable my lstm layer like this
```
class model(nn.Module):
    def __init__(self, insize=32, hiddensize = 16, level=10):
        super(model, self).__init__()

        self.conv = nn.Conv2d(1, 32, kernel_size=(1,20))
        #self.lstm = nn.LSTM(input_size=insize, hidden_size=hiddensize, batch_first=True)
        self.fc1 = nn.Linear(in_features=hiddensize, out_features=3)

    def forward(self, x):
        out = self.conv(x)
        
        out = out.permute(0, 2, 1, 3)
        out = out.squeeze(3)
        #out, _ = self.lstm(out)

        out = torch.narrow(out, 1, out.shape[1]-1, 1).squeeze(1)
        
        out = self.fc1(out)

        return out
```
Everything worked fine, no more error happened.
I wonder why the error only happened when I have lstm layer

### Versions

```
Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.7.1
[pip3] flake8==3.9.2
[pip3] focal-loss-torch==0.1.2
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.1.0
[pip3] onnx==1.11.0
[pip3] onnx-simplifier==0.3.10
[pip3] onnxoptimizer==0.2.7
[pip3] onnxruntime==1.11.1
[pip3] pytorchcv==0.0.67
[pip3] torch==1.13.0+cu116
[pip3] torch-tb-profiler==0.4.1
[pip3] torch2trt==0.3.0
[pip3] torchaudio==0.13.0+cu116
[pip3] torchcv==0.0.2
[pip3] torchinfo==1.6.3
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.14.0+cu116
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] efficientnet-pytorch      0.7.1                    pypi_0    pypi
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] focal-loss-torch          0.1.2                    pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.20.3           py39hf144106_0
[conda] numpy-base                1.20.3           py39h74d4b33_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorchcv                 0.0.67                   pypi_0    pypi
[conda] torch                     1.13.0+cu116             pypi_0    pypi
[conda] torch-tb-profiler         0.4.1                    pypi_0    pypi
[conda] torch2trt                 0.3.0                    pypi_0    pypi
[conda] torchaudio                0.13.0+cu116             pypi_0    pypi
[conda] torchcv                   0.0.2                    pypi_0    pypi
[conda] torchinfo                 1.6.3                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.14.0+cu116             pypi_0    pypi
```

cc @malfet @seemethere @csarofeen @ptrblck @xwang233