Repository: pytorch
Issue: Incorrect output loss value under specific CUDA version
Labels: high priority,module: dependency bug,module: numerical-stability,module: cuda,triaged,module: linear algebra,module: tf32

## 🐛 Bug

I ran the example in https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-tensors-and-autograd and I got output loss always between ~0 and ~20 with CUDA 11.0. I switched to CUDA 10.2, the output loss seemed to be correct (<1e-7). I tried different num of iterations up to 100,000 and I got the same result.

## To Reproduce

Steps to reproduce the behavior:

1. Run the code copied from https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-tensors-and-autograd. I set the num of iterations to 10000.

```
# -*- coding: utf-8 -*-
import torch

dtype = torch.float
# device = torch.device("cpu")
device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
# Setting requires_grad=False indicates that we do not need to compute gradients
# with respect to these Tensors during the backward pass.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(10000):
    # Forward pass: compute predicted y using operations on Tensors; these
    # are exactly the same operations we used to compute the forward pass using
    # Tensors, but we do not need to keep references to intermediate values since
    # we are not implementing the backward pass by hand.
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # Compute and print loss using operations on Tensors.
    # Now loss is a Tensor of shape (1,)
    # loss.item() gets the scalar value held in the loss.
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass. This call will compute the
    # gradient of loss with respect to all Tensors with requires_grad=True.
    # After this call w1.grad and w2.grad will be Tensors holding the gradient
    # of the loss with respect to w1 and w2 respectively.
    loss.backward()

    # Manually update weights using gradient descent. Wrap in torch.no_grad()
    # because weights have requires_grad=True, but we don't need to track this
    # in autograd.
    # An alternative way is to operate on weight.data and weight.grad.data.
    # Recall that tensor.data gives a tensor that shares the storage with
    # tensor, but doesn't track history.
    # You can also use torch.optim.SGD to achieve this.
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()
```

Fractions of the output:

```
99 757.3246459960938
199 15.870492935180664
299 12.101653099060059
399 9.181726455688477
499 5.7796196937561035
599 9.187292098999023
699 8.796398162841797
799 8.489400863647461
899 8.904274940490723
999 5.577568054199219
1099 7.905296325683594
......
9099 6.956545352935791
9199 12.90721321105957
9299 9.553459167480469
9399 8.973970413208008
9499 9.481210708618164
9599 6.536320686340332
9699 8.174590110778809
9799 11.91832160949707
9899 7.639415264129639
```

## Expected behavior

This is the output I got on another server. This seems to be correct.

```
99 1139.0550537109375
199 18.11379051208496
299 0.5359313488006592
399 0.019708311185240746
499 0.0011226845672354102
599 0.0001853789872257039
699 6.25241082161665e-05
799 3.071533137699589e-05
899 1.884384619188495e-05
999 1.285602502321126e-05
1099 9.616760507924482e-06
......
9099 2.8602207180483674e-07
9199 2.7375682520869304e-07
9299 2.727583989781124e-07
9399 2.7066275265497097e-07
9499 2.706639747884765e-07
9599 2.6544344677859044e-07
9699 2.6234039296468836e-07
9799 2.550268334289285e-07
9899 2.47128951968989e-07
9999 2.423165028631047e-07
```

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

Output:

```
Collecting environment information...
PyTorch version: 1.7.0+cu110
Is debug build: True
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-6ubuntu2) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 3090

Nvidia driver version: 455.23.05
cuDNN version: Probably one of the following:
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] torch==1.7.0+cu110
[pip3] torchaudio==0.7.0
[pip3] torchvision==0.8.1+cu110
[conda] cudatoolkit               11.0.221             h6bb024c_0    defaults
[conda] numpy                     1.19.4                   pypi_0    pypi
[conda] torch                     1.7.0+cu110              pypi_0    pypi
[conda] torchaudio                0.7.0                    pypi_0    pypi
[conda] torchvision               0.8.1+cu110              pypi_0    pypi
```

## Additional context




cc @ezyang @gchanan @zou3519 @bdhirsh @ngimel @vishwakftw @jianyuh @nikitaved @pearu @mruberry @heitorschueroff @zasdfgbnm