Repository: pytorch
Issue: Can't reproduce/non-deterministic results with CUDA
Labels: module: cuda,triaged,module: determinism

### 🐛 Describe the bug

Hi! I'm trying to reproduce results with a minimal text classifier using suggestions from [PyTorch — Randomness](https://pytorch.org/docs/stable/notes/randomness.html), but I still get different final weights with each launch. Could you please have a look at it?
My environment is Python 3.9, torch 2.0.1 with CUDA 18.1 ([this one](
https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp39-cp39-linux_x86_64.whl)), Ubuntu 18.04 (x64, WSL), RTX 2080 Ti. I also set `CUBLAS_WORKSPACE_CONFIG=":4096:8"`.

```
import torch

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

import random
from torch import nn
from torch.nn import functional as F
import numpy as np
from sklearn.utils import gen_batches


VOCAB_SIZE = 50000
NUM_CLASSES = 20
NUM_SAMPLES = 1000
DEFAULT_LEN = 64


class CNN(nn.Module):
    def __init__(
        self,
        dim=300,
        dropout_keep_p=0.9,
        lr=1e-3,
        n_filt=64,
        filt_sizes=[1, 1, 2, 2],
    ):
        super().__init__()

        self.lr = lr
        self.embedding = nn.Embedding(VOCAB_SIZE, dim, padding_idx=0, max_norm=5.0)
        self.convs = nn.ModuleList(
            [
                nn.Conv2d(
                    in_channels=1,
                    out_channels=n_filt,
                    kernel_size=(fs, dim),
                )
                for fs in filt_sizes
            ]
        )
        self.dropout = nn.Dropout(1 - dropout_keep_p)
        self.fc = nn.Linear(len(filt_sizes) * n_filt, NUM_CLASSES)
        self.init_weights()

    def init_weights(self):
        self.embedding.weight.data.uniform_(-0.5, 0.5)
        for conv in self.convs:
            conv.weight.data.uniform_(-0.5, 0.5)
        self.fc.weight.data.uniform_(-0.5, 0.5)
        self.fc.bias.data.zero_()

    def forward(self, input):
        embedded = self.embedding(input)
        embedded = embedded.unsqueeze(1)
        conv_output = [F.silu(conv(embedded)).squeeze(-1) for conv in self.convs]
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(-1) for conv in conv_output]
        cat = self.dropout(torch.cat(pooled, dim=1))
        logits = self.fc(cat)
        return logits


if __name__ == "__main__":
    seed = 0
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    criterion = nn.CrossEntropyLoss(reduction="none")

    model = CNN()
    model.to("cuda")

    optimizer = torch.optim.Adam(
        model.parameters(), lr=float(model.lr)
    )

    def train(sample_batches, label_batches):
        for sample_batch, label_batch in zip(sample_batches, label_batches):
            sample_batch = sample_batch.to("cuda")
            label_batch = label_batch.to("cuda")
            optimizer.zero_grad()
            output = model(sample_batch)
            loss = criterion(output, label_batch)
            loss = loss.mean()
            loss.backward()
            optimizer.step()

    gen = np.random.default_rng(seed=0)

    samples = [
        gen.integers(low=0, high=VOCAB_SIZE, size=DEFAULT_LEN)
        for _ in range(NUM_SAMPLES)
    ]
    labels = [gen.integers(low=0, high=NUM_CLASSES) for i in range(NUM_SAMPLES)]

    print(
        "init checksum",
        torch.sum([p.data for n, p in model.named_parameters()][0]).item(),
    )

    slices = list(gen_batches(n=len(samples), batch_size=64, min_batch_size=64))
    sample_batches = [torch.as_tensor(samples[slice]) for slice in slices]
    label_batches = [torch.as_tensor(labels[slice]) for slice in slices]

    model.train()

    for i in range(10):
        train(sample_batches, label_batches)

    print(
        "train checksum",
        torch.sum([p.data for n, p in model.named_parameters()][0]).item(),
    )
```

### Versions

```
PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.26.3
Libc version: glibc-2.27

Python version: 3.9.16 (main, Mar  8 2023, 14:00:05)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2080 Ti
Nvidia driver version: 531.41
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
Stepping:            9
CPU MHz:             2998.313
BogoMIPS:            5996.62
Hypervisor vendor:   Microsoft
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1+cu118
[pip3] triton==2.0.0
[conda] numpy                     1.24.3                   pypi_0    pypi
[conda] torch                     2.0.1+cu118              pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi
```

cc @ngimel @mruberry @kurtamohler