Repository: pytorch
Issue: Forward results vary depending on batch size on A100 machine
Labels: module: cuda,module: convolution,triaged,module: numerical-reproducibility

## 🐛 Bug

Hi pytorch teams,
When increasing the inference batch size on A100 machine, the model output will differ than that of batch size = 1.
I simplify the model to only one torch.nn.Conv2d layer, this problem still occur.

## To Reproduce

```python
import torch
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(0)
torch.cuda.manual_seed_all(0)

conv = torch.nn.Conv2d(100, 100, 3)

x2 = torch.rand(2, 100, 256, 256)
x1 = x2.clone()[:1]

conv.cuda()
x2 = x2.cuda()
x1 = x1.cuda()

with torch.no_grad():
    out2 = conv(x2)
    out1 = conv(x1)

print((out2[:1] == out1).all())

print(torch.allclose(out2[:1], out1))
```
![image](https://user-images.githubusercontent.com/35265996/138380244-87d839e9-13e6-4159-aa00-c6268ea48112.png)

## Expected behavior

out2[:1] and out1 should be the same, or at least all close.

## Environment

Collecting environment information...
PyTorch version: 1.7.0a0+7036e91
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.14.0
Libc version: glibc-2.9

Python version: 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-65-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.1.74
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM-80GB
GPU 1: NVIDIA A100-SXM-80GB
GPU 2: NVIDIA A100-SXM-80GB
GPU 3: NVIDIA A100-SXM-80GB
GPU 4: NVIDIA A100-SXM-80GB
GPU 5: NVIDIA A100-SXM-80GB
GPU 6: NVIDIA A100-SXM-80GB
GPU 7: NVIDIA A100-SXM-80GB

Nvidia driver version: 470.57.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] pytorch-transformers==1.1.0
[pip3] torch==1.7.0a0+7036e91
[pip3] torchtext==0.8.0a0
[pip3] torchvision==0.8.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.1                      144  
[conda] mkl-include               2019.1                      144  
[conda] nomkl                     3.0                           0  
[conda] numpy                     1.19.1           py36h30dfecb_0  
[conda] numpy-base                1.19.1           py36h75fe3a5_0  
[conda] pytorch-transformers      1.1.0                    pypi_0    pypi
[conda] torch                     1.7.0a0+7036e91          pypi_0    pypi
[conda] torchtext                 0.8.0a0                  pypi_0    pypi
[conda] torchvision               0.8.0a0                  pypi_0    pypi

## Additional context

My another machine with V100 GPU works correctly as expected.

p.s. How can I debug such things?
Thanks!

cc @ngimel