Repository: pytorch
Issue: Anomaly detection: Error detected in CudnnRnnBackward0
Labels: module: cudnn,module: rnn,module: cuda,triaged

## 🐛 Bug

## To Reproduce

When I run `output, _ = self.gru(output)` I get the following traceback:

```
[W ..\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in CudnnRnnBackward0. Traceback of forward call that caused the error:
  File "C:\Users\Gili\AppData\Roaming\JetBrains\IntelliJIdea2021.2\plugins\python\helpers\pydev\pydevd.py", line 2173, in <module>
    main()
  File "C:\Users\Gili\AppData\Roaming\JetBrains\IntelliJIdea2021.2\plugins\python\helpers\pydev\pydevd.py", line 2164, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Users\Gili\AppData\Roaming\JetBrains\IntelliJIdea2021.2\plugins\python\helpers\pydev\pydevd.py", line 1476, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "C:\Users\Gili\AppData\Roaming\JetBrains\IntelliJIdea2021.2\plugins\python\helpers\pydev\pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\Gili\AppData\Roaming\JetBrains\IntelliJIdea2021.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 785, in <module>
    main()
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 771, in main
    tune_hyperparameters(java_dataset)
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 743, in tune_hyperparameters
    study.optimize(lambda trial: optimize_train(trial, java_dataset),
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\study.py", line 400, in optimize
    _optimize(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 66, in _optimize
    _optimize_sequential(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 163, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\optuna\study\_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 743, in <lambda>
    study.optimize(lambda trial: optimize_train(trial, java_dataset),
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 708, in optimize_train
    return train(dataset, learning_rate, max_epochs, seq2seq_type, seq2seq_layers, linear_layers,
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 492, in train
    trainer.fit(model)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 552, in fit
    self._run(model)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 917, in _run
    self._dispatch()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 985, in _dispatch
    self.accelerator.start_training(self)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 995, in run_stage
    return self._run_train()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1044, in _run_train
    self.fit_loop.run()
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 130, in advance
    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 100, in run
    super().run(batch, batch_idx, dataloader_idx)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 147, in advance
    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 201, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 395, in _optimizer_step
    model_ref.optimizer_step(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\core\lightning.py", line 1616, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\core\optimizer.py", line 206, in step
    self.__optimizer_step(closure=closure, profiler_name=profiler_name, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\core\optimizer.py", line 128, in __optimizer_step
    trainer.accelerator.optimizer_step(self._optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 292, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\plugins\precision\native_amp.py", line 57, in pre_optimizer_step
    result = lambda_closure()  # native amp does not support closures
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 235, in _training_step_and_backward_closure
    result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 536, in training_step_and_backward
    result = self._training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\loops\batch\training_batch_loop.py", line 306, in _training_step
    training_step_output = self.trainer.accelerator.training_step(step_kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 193, in training_step
    return self.training_type_plugin.training_step(*step_kwargs.values())
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 172, in training_step
    return self.model.training_step(*args, **kwargs)
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 308, in training_step
    actual = self(input)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:/Users/Gili/Documents/myproject/aggregator/src/main/python/com.mycompany.ai/predict_outdoor_temperature.py", line 262, in forward
    output, _ = self.gru(output)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Gili\Documents\myproject\python\lib\site-packages\torch\nn\modules\rnn.py", line 849, in forward
    result = _VF.gru(input, hx, self._flat_weights, self.bias, self.num_layers,
 (function _print_stack)
```

Unfortunately, I don't have a minimal testcase to share with you but feel free to ask me for any more information you need.

## Environment

```
PyTorch version: 1.10.0.dev20210918+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19043-SP0
Is CUDA available: True
CUDA runtime version: 11.4.120
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 471.96
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] pytorch-lightning==1.4.7
[pip3] torch==1.10.0.dev20210918+cu113
[pip3] torch-tb-profiler==0.2.1
[pip3] torchmetrics==0.5.1
[pip3] torchvision==0.11.0.dev20210918+cu113
[conda] Could not collect
```

cc @csarofeen @ptrblck @xwang233 @zou3519 @ngimel