Repository: pytorch
Issue: `with torch.backends.cudnn.flags(deterministic=True)` doesn't give an exception for ctc_loss backward on CUDA
Labels: module: cudnn,triaged,module: determinism

### 🐛 Describe the bug

Hi! I noticed some strange behavior of `with torch.backends.cudnn.flags(deterministic=True)` for ctc loss backward on CUDA.

Main problem is that `with torch.backends.cudnn.flags(deterministic=True)` doesn't give an exception while `torch.use_deterministic_algorithms(True)` does.

## Reproduction

I maid two scripts to reproduce bug.

### torch.use_deterministic_algorithms(True) - Exception

```
import torch
from torch import nn
import torch.nn.functional as F

torch.use_deterministic_algorithms(True)

device = torch.device('cuda') 

T = 50      # Input sequence length
C = 20      # Number of classes (including blank)
N = 16      # Batch size
S = 30      # Target sequence length of longest target in batch (padding length)
S_min = 10  # Minimum target length, for demonstration purposes

# Initialize random batch of input vectors, for *size = (T,N,C)
input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_().to(device)

# Initialize random batch of targets (0 = blank, 1:C = classes)
target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long).to(device)

input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long).to(device)
target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long).to(device)

loss = F.ctc_loss(input, target, input_lengths, target_lengths)

loss.backward()
```

And I get following exception:

```
RuntimeError: ctc_loss_backward_gpu does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
```

### with torch.backends.cudnn.flags(deterministic=True) - No exception

```
import torch
import torch.nn.functional as F

device = torch.device('cuda') 

T = 50      # Input sequence length
C = 20      # Number of classes (including blank)
N = 16      # Batch size
S = 30      # Target sequence length of longest target in batch (padding length)
S_min = 10  # Minimum target length, for demonstration purposes

# Initialize random batch of input vectors, for *size = (T,N,C)
input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_().to(device)

# Initialize random batch of targets (0 = blank, 1:C = classes)
target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long).to(device)

input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long).to(device)
target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long).to(device)


with torch.backends.cudnn.flags(deterministic=True):
    loss = F.ctc_loss(input, target, input_lengths, target_lengths)


loss.backward()
```

And this script runs without exceptions. And I found it strange, because I do the same. I also tried on colab and got two exceptions.

Example of usage - https://github.com/espnet/espnet/blob/5fa6dcc4e649dc66397c629d0030d09ecef36b80/espnet/nets/pytorch_backend/ctc.py#L65

### Versions

```
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.1 (default, Dec 11 2020, 14:32:07)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB

Nvidia driver version: 470.82.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] pytorch-lightning==1.5.4
[pip3] torch==1.9.0
[pip3] torch-stft==0.1.4
[pip3] torchaudio==0.9.0a0+33b2469
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py39h27cfd23_1  
[conda] mkl_fft                   1.3.0            py39h42c9631_2  
[conda] mkl_random                1.2.1            py39ha9443f7_2  
[conda] numpy                     1.20.2           py39h2d18471_0  
[conda] numpy-base                1.20.2           py39hfae3a4d_0  
[conda] pytorch                   1.9.0           py3.9_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-lightning         1.5.4                    pypi_0    pypi
[conda] torch-stft                0.1.4                    pypi_0    pypi
[conda] torchaudio                0.9.0                      py39    pytorch
[conda] torchmetrics              0.6.0                    pypi_0    pypi
[conda] torchvision               0.10.0               py39_cu111    pytorch
```

`torch.backends.cudnn.version()` - `8005`

cc @csarofeen @ptrblck @xwang233 @mruberry @kurtamohler