Repository: pytorch
Issue: torch.rand(...) is not consistent for large shape dimensions across GPUs (with the same random seed)
Labels: module: cuda,triaged,module: random

### 🐛 Describe the bug

Generating a random tensor with `torch.rand` yields different results on different GPUs once dimensions are starting to become large.

The following simple Python code:

```python
import torch 

generator = torch.Generator(device='cuda').manual_seed(0)
print(torch.rand((8, 77, 1024), device='cuda', generator=generator).abs().sum())
```

yields different results depending on the GPU one is working with. This poses a quite significant problem for testing where applications start with gaussian noise of higher dimensions such as https://github.com/huggingface/diffusers

The interesting part is that one does get the same results for smaller dimensions such as `(8, 77, 10)`

### Versions

First is GPU is a TITAN RTX with the following version:
```
PyTorch version: 1.12.1+cu102                                                                                          
Is debug build: False                                                                                                  
CUDA used to build PyTorch: 10.2                                                                                       
ROCM used to build PyTorch: N/A                                                                                        
                                                                                                                       
OS: Ubuntu 19.10 (x86_64)                                                                                              
GCC version: (Ubuntu 8.4.0-1ubuntu1~19.10) 8.4.0                                                                       
Clang version: Could not collect                                                                                       
CMake version: version 3.24.1                              
Libc version: glibc-2.30                                                                                               
                                                                                                                                        
Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)                                  
Python platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17                                                          
Is CUDA available: True                                                                                                
CUDA runtime version: Could not collect                                                                                                                                                                                                       
CUDA_MODULE_LOADING set to:                                                                                            
GPU models and configuration:                                                                                                           
GPU 0: NVIDIA TITAN RTX                                                                                                
GPU 1: NVIDIA TITAN RTX                                                                                                                                                                                                                       
                                                                                                                       
Nvidia driver version: 510.47.03                           
cuDNN version: Probably one of the following:                                                                          
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5                                                                            
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7                                                            
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7                                                            
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn.so.8                                                            
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8                                                  
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_train.so.8                                                  
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8                                                  
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8                                                  
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8                                                  
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_train.so.8                                                  
HIP runtime version: N/A                                                                                               
MIOpen runtime version: N/A                                                                                            
Is XNNPACK available: True                                                                                             
                                                                                                                       
Versions of relevant libraries:                                                                                        
[pip3] mypy-extensions==0.4.3                                                                                          
[pip3] numpy==1.23.2                                                                                                   
[pip3] open-clip-torch==1.3.0                                                                                          
[pip3] pytorch-fid==0.2.1                                                                                              
[pip3] pytorch-lightning==1.6.4                                                                                                         
[pip3] torch==1.12.1                                                                                                                    
[pip3] torchaudio==0.11.0                                                                                                               
[pip3] torchmetrics==0.9.1                                                                                                              
[pip3] torchvision==0.12.0                                                                                                              
[pip3] v-diffusion-pytorch==0.0.1                                                                                                       
[conda] _pytorch_select           0.1                       cpu_0                                                                       
[conda] blas                      1.0                         mkl                                                                       
[conda] cudatoolkit               11.3.1               h2bc3f7f_2                                                                       
[conda] libmklml                  2019.0.5             h06a4308_0                                                                       
[conda] mkl                       2020.2                      256                                                                       
[conda] mkl-service               2.3.0            py38he904b0f_0                                                                       
[conda] mkl_fft                   1.3.0            py38h54f3939_0                                                                                             
[conda] mkl_random                1.1.1            py38h0573a6f_0                                                                                             
[conda] numpy                     1.23.2                   pypi_0    pypi                                                                                     
[conda] numpy-base                1.19.2           py38hfa32c7d_0                                                                                             
[conda] open-clip-torch           1.3.0                     dev_0    <develop>                                                                                
[conda] pytorch-fid               0.2.1                    pypi_0    pypi                                                                                     
[conda] pytorch-lightning         1.6.4                    pypi_0    pypi                                                                                     
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly                                                                          
[conda] torch                     1.12.1                   pypi_0    pypi                                                                                     
[conda] torchaudio                0.10.1+cu113             pypi_0    pypi                                                                                     
[conda] torchmetrics              0.9.1                    pypi_0    pypi                                                                                     
[conda] torchvision               0.12.0               py38_cu113    pytorch                                                                                  
[conda] v-diffusion-pytorch       0.0.1                    pypi_0    pypi  
```

and second is a V100 with the following version:

```
Collecting environment information...
PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (Debian 8.3.0-6) 8.3.0
Clang version: Could not collect
CMake version: version 3.13.4
Libc version: glibc-2.10

Python version: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.13
Is CUDA available: True
CUDA runtime version: 11.0.221
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 460.73.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.6
[pip3] pytorch-lightning==1.7.7
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchmetrics==0.10.1
[pip3] torchvision==0.13.1+cu113
[conda] blas                      2.113                       mkl    conda-forge
[conda] blas-devel                3.9.0            13_linux64_mkl    conda-forge
[conda] cudatoolkit               11.1.1              h6406543_10    conda-forge
[conda] dlenv-pytorch-1-10-gpu    1.0.20220226     py37h0ee201a_0    file:///tmp/conda-pkgs
[conda] libblas                   3.9.0            13_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            13_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            13_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            13_linux64_mkl    conda-forge
[conda] mkl                       2022.0.1           h8d4b97c_803    conda-forge
[conda] mkl-devel                 2022.0.1           ha770c72_804    conda-forge
[conda] mkl-include               2022.0.1           h8d4b97c_803    conda-forge
[conda] numpy                     1.19.5           py37h038b26d_2    conda-forge
[conda] pytorch                   1.10.0          py3.7_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.11.1+cu111             pypi_0    pypi
```

cc @ngimel @pbelevich