Repository: pytorch
Issue: cuDNN error upon the backward method with a sliced tensor output from nn.GRU.
Labels: module: cudnn,module: rnn,module: cuda,triaged

## 🐛 Bug

Hi. I encountered "RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED" while attempting to run the backward method in my model. After some examinations, I figured out that the error originates from the sliced tensor output from a nn.GRU instance in my model; all downstream tensors that depend on it would fail to run the backward method.

The problem happens only if I use nn.GRU. If I use other RNN variants, for example nn.LSTM, the error disappears. Also, the problem happens only if I set the hidden state of the GRU to be a NON-multiple of 4. The following is a minimal nn.Module definition and codelines to reproduce the situation.
```
import torch
import torch.nn as nn


class testmodel(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(testmodel, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        
        self.lstm = nn.LSTM(input_size=self.input_size,
                            hidden_size=self.hidden_size,
                            num_layers=1)
        
        self.gru = nn.GRU(input_size=self.input_size,
                          hidden_size=self.hidden_size,
                          num_layers=1)
   
    def forward(self, sequence):
        time = sequence.size(0)
        batch = sequence.size(1)
        
        # Run GRU
        feature, _ = self.gru(sequence)

        # Running LSTM instead would not produce the error.
        # feature, _ = self.lstm(sequence)
        
        output = feature[..., :3] # Slice the RNN output.

        return output

time = 3
batch_size = 3
input_size = 3

device = torch.device("cuda")
sequence = torch.normal(0.0, 1.0, size=(time, batch_size, input_size)).to(device)

succeed = set()
for hidden_size in range(4, 33):
    model = testmodel(input_size, hidden_size).to(device)
    res = model(sequence)

    loss = (res + 1).sum() # Run any downstream operations.
    
    # Collect the hidden_size settings that do not encounter the error.
    try:
        loss.backward()
        succeed.add(hidden_size)
    
    except Exception as e:
        error = e
        pass

print("hidden_states settings with no problem: {}.".format(succeed))
print("All other hidden_states settings in [4, 32] failed.")
print(error)
```

## To Reproduce

Steps to reproduce the behavior:

1. Initialize a nn.GRU instance with its hidden state size set to be a non-multiple of 4 (e.g., 6, 7, 11, etc.).
2. Run the GRU with an input of shape (time, batch, input_size) and get the output of shape (time, batch, hidden_size).
3. Slice the output to the shape (time, batch, slice_size)
4. (Do any downstream operations with the slice and) run the backward method.

**Error Traceback**
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-73aecda8025d> in <module>
      3 loss = (res + 1).sum()
      4 
----> 5 loss.backward()

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--> 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

~/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    148 
    149 

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
```

## Expected behavior

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED is raised.

## Environment

PyTorch version: 1.8.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.10.2

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: TITAN Xp
GPU 2: TITAN Xp
GPU 3: TITAN Xp

Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] msgpack-numpy==0.4.4.2
[pip3] numpy==1.16.4
[pip3] numpy-quaternion==2019.10.3.10.26.21
[pip3] numpydoc==0.8.0
[pip3] torch==1.8.0+cu111
[pip3] torchaudio==0.8.0
[pip3] torchvision==0.9.0+cu111
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.0.130                      0
[conda] mkl                       2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] msgpack-numpy             0.4.4.2                  pypi_0    pypi
[conda] numpy                     1.16.3                   pypi_0    pypi
[conda] numpy-base                1.15.4           py37hde5b4d6_0
[conda] numpy-quaternion          2019.10.3.10.26.21          pypi_0    pypi
[conda] numpydoc                  0.8.0                    py37_0
[conda] torch                     1.8.0+cu111              pypi_0    pypi
[conda] torchaudio                0.8.0                    pypi_0    pypi
[conda] torchvision               0.9.0+cu111              pypi_0    pypi

## Additional context
N/A


cc @ngimel @csarofeen @ptrblck @xwang233 @zou3519