Repository: pytorch
Issue: different results on each batch_size in torch==1.10.2+cu113 on RTX 3080
Labels: triaged,module: numerical-reproducibility

### 🐛 Describe the bug

on each batch size, the result is different, I tried on google-colab(with there default pytorch) the outputs were consistent.

```
from PIL import Image
import requests
import torch
import os


from transformers import CLIPProcessor, CLIPModel, CLIPVisionModel


urls = ['http://images.cocodataset.org/val2017/000000039769.jpg' , 'http://images.cocodataset.org/test-stuff2017/000000000448.jpg','http://images.cocodataset.org/test-stuff2017/000000000128.jpg']
get_image = lambda x: Image.open(requests.get(x, stream=True).raw)

images = [get_image(i) for i in urls]
images = images * 10
torch.use_deterministic_algorithms(True)

model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32").to("cuda").eval()
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
inputs = processor(images=images, return_tensors="pt")

with torch.no_grad():
      output_clip_images = model(**inputs.to("cuda"))


inputs = processor(images=images[0], return_tensors="pt")
with torch.no_grad():
      output_clip_image = model(**inputs.to("cuda"))
close = torch.isclose(output_clip_image[1][0], output_clip_images[1][0])
close.sum(), close.shape
```
 > (tensor(11, device='cuda:0'), torch.Size([768]))

the same code on google colab but with `torch.use_deterministic_algorithms(False)` because it raises an error 
gives:
> (tensor(768, device='cuda:0'), torch.Size([768]))

```
np.testing.assert_almost_equal(output_clip_image[1][0].cpu().detach().numpy(), output_clip_images[1][0].cpu().detach().numpy())
```

```

AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 768 / 768 (100%)
Max absolute difference: 0.00241792
Max relative difference: 0.5413908
 x: array([-3.5340825e-01,  4.3769903e-02, -7.5632817e-01, -8.6524528e-01,
        5.1617998e-01, -8.6147100e-01,  3.0487695e-01,  1.0068034e+00,
        4.8502433e-01,  4.2452109e-01,  2.1260881e+00, -6.3165623e-01,...
 y: array([-3.5306537e-01,  4.2240802e-02, -7.5647628e-01, -8.6631018e-01,
        5.1585835e-01, -8.6233985e-01,  3.0509090e-01,  1.0060844e+00,
        4.8487094e-01,  4.2465281e-01,  2.1263967e+00, -6.2989092e-01,...

```


### Versions

```
Collecting environment information...
PyTorch version: 1.10.2+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: elementary OS 6.1 Jólnir (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.29

Python version: 3.7.9 (default, Feb 20 2022, 21:45:03)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.13.0-30-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] torch==1.10.2+cu113
[pip3] torchaudio==0.10.2+cu113
[pip3] torchvision==0.11.3+cu113
[conda] Could not collect


```