Repository: pytorch
Issue: LSTM built-in dropout not reproducible on GPU
Labels: module: cudnn,module: nn,triaged,module: random,module: determinism

### 🐛 Describe the bug

The results of a LSTM with the built-in dropout aren't reproducible.    
**Test 1 for built-in dropout:**   
```python
import torch
seed = 42
ex = torch.ones(10,2).cuda()
lstm = torch.nn.LSTM(input_size=2, hidden_size=4, num_layers=2, dropout=0.5, batch_first=True).cuda()

# set seed
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

# check dropout output
print("1. lstm with dropout:")
print(lstm(ex))

# save seed states
saved_torch_seed_state = torch.get_rng_state()
saved_torch_cuda_seed_state = torch.cuda.get_rng_state_all()

# set seed again
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

# check dropout output again
print("1. lstm with dropout after re-seeding:")
print(lstm(ex))

# check second output
print("2. lstm with dropout after re-seeding:")
print(lstm(ex))

# set old seed states
torch.set_rng_state(saved_torch_seed_state)
torch.cuda.set_rng_state_all(saved_torch_cuda_seed_state)

# check if the second output here is the same as the second one above
print("2. lstm with dropout after setting rng states:")
print(lstm(ex))
```
**Expected behaviour:** The results of the second LSTM run with dropout after re-seeding and of the second LSTM run with dropout after setting rng states should be identical.   
The results are identical when using the CPU, but not the GPU. I tried to set ``torch.backends.cudnn.benchmark = False``, ``torch.use_deterministic_algorithms(True)``, ``torch.backends.cudnn.deterministic = True``, ``os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:2"`` and/or ``os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"``, but nothing worked. The workaround I found is to use first an LSTM with one layer without dropout, then a dropout layer, then an LSTM with one layer again, essentially mimicking the LSTM with two layers and built-in dropout. The results are reproducible.   
    
**Test 2 for independent dropout layer:**
```python
import torch
seed = 42
ex = torch.ones(10,2).cuda()
lstm = torch.nn.LSTM(input_size=2, hidden_size=4, num_layers=1, batch_first=True).cuda()
drop = torch.nn.Dropout(0.5).cuda()
lstm2 = torch.nn.LSTM(input_size=4, hidden_size=4, num_layers=1, batch_first=True).cuda()

# set seed
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

# check dropout output
print("1. lstm with dropout:")
x,_ = lstm(ex)
x = drop(x)
print(lstm2(x))

# save seed states
saved_torch_seed_state = torch.get_rng_state()
saved_torch_cuda_seed_state = torch.cuda.get_rng_state_all()

# set seed again
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

# check dropout output again
print("1. lstm with dropout after re-seeding:")
x,_ = lstm(ex)
x = drop(x)
print(lstm2(x))

# check second output
print("2. lstm with dropout after re-seeding:")
x,_ = lstm(ex)
x = drop(x)
print(lstm2(x))

# set old seed states
torch.set_rng_state(saved_torch_seed_state)
torch.cuda.set_rng_state_all(saved_torch_cuda_seed_state)

# check if the second output here is the same as the second one above
print("2. lstm with dropout after setting rng states:")
x,_ = lstm(ex)
x = drop(x)
print(lstm2(x))
```

### Versions

(Output of ``collect_env.py``, slightly edited)

PyTorch version: 2.0.0+cu117   
Is debug build: False   
CUDA used to build PyTorch: 11.7   
ROCM used to build PyTorch: N/A   
     
OS: Ubuntu 20.04.6 LTS (x86_64), error also occurs on Windows and Centos7  
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0    
Clang version: Could not collect   
CMake version: version 3.26.3   
Libc version: glibc-2.31    

Python version: 3.8.10 (default, Mar 13 2023, 10:26:41)  [GCC 9.4.0] (64-bit runtime)    
Python platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29    
Is CUDA available: True   
CUDA runtime version: 11.7.64    
CUDA_MODULE_LOADING set to: LAZY    
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650   
Nvidia driver version: 531.79    
cuDNN version: 8.8.1    
HIP runtime version: N/A   
MIOpen runtime version: N/A   
Is XNNPACK available: True   

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
Vendor ID:                       AuthenticAMD
CPU family:                      25
Model:                           80
Model name:                      AMD Ryzen 5 5600H with Radeon Graphics
Stepping:                        0
CPU MHz:                         3293.653
BogoMIPS:                        6587.30
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       192 KiB
L1i cache:                       192 KiB
L2 cache:                        3 MiB
L3 cache:                        16 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat umip vaes vpclmulqdq rdpid fsrm

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] pytorch-lightning==1.8.6
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchmetrics==0.7.3
[pip3] torchvision==0.15.1
[pip3] triton==2.0.0

cc @csarofeen @ptrblck @xwang233 @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @pbelevich @kurtamohler