Repository: pytorch
Issue: [BUG] Inconsistent initialization on different machines (tensor.uniform_())
Labels: triaged,module: random

## 🐛 Bug
Instantiating the same model on different machine give different initial weights.
I tracked down the problem to be in the implementation of tensor.uniform_() which generate slightly different distribution on different machines.

A related issue is that the generated distribution will be different if generated using CPU or GPU (on the same machine)

## To Reproduce
Here is some code samples to reproduce the behavior.

CPU version (Should be run on 2 different machines) :
```python
import torch
import random
import numpy as np

seed = 42

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)

torch.use_deterministic_algorithms(True)

min_val = -99999
max_val = 99999
dim = (146,520,4)
for i in range(10):
    tensor = torch.ones(dim)
    with torch.no_grad():
        tensor.uniform_(min_val, max_val)

    print(f"Mean : {tensor.mean()}  std : {tensor.std()}")
```

Result on machine 1 : 
```
Mean : -1.7765913009643555  std : 57748.234375
Mean : -42.21729278564453  std : 57701.90625
Mean : -181.7095947265625  std : 57723.19140625
Mean : -122.64220428466797  std : 57710.4375
Mean : 25.23169708251953  std : 57750.62109375
Mean : -77.48368835449219  std : 57703.703125
Mean : 46.812171936035156  std : 57790.01953125
Mean : -198.31365966796875  std : 57640.6328125
Mean : 7.587873935699463  std : 57819.83984375
Mean : 44.30927658081055  std : 57649.36328125
```

Result on machine 2 : 
```
Mean : -1.776593804359436  std : 57748.234375
Mean : -42.21727752685547  std : 57701.90625
Mean : -181.7095947265625  std : 57723.19140625
Mean : -122.64220428466797  std : 57710.4375
Mean : 25.231721878051758  std : 57750.62109375
Mean : -77.4836654663086  std : 57703.703125
Mean : 46.81214904785156  std : 57790.01953125
Mean : -198.3136749267578  std : 57640.6328125
Mean : 7.587862014770508  std : 57819.83984375
Mean : 44.30925369262695  std : 57649.36328125
```

You can see that the mean is slightly different (Look after fifth decimal)

Same behavior can be observed when using the GPU with the following code : 
```python
import torch
import random
import numpy as np

seed = 42

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)

if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
else:
    print('[ERROR] this test is meant to be run on gpu')
    exit(1)

torch.use_deterministic_algorithms(True)

device = 'cuda:0'
min_val = -99999
max_val = 99999
dim = (146, 520, 4)
for i in range(10):
    tensor = torch.ones(dim).to(device)
    with torch.no_grad():
        tensor.uniform_(min_val, max_val)

    print(f"Mean : {tensor.mean()}  std : {tensor.std()}")
```

The result on the same machine are however vastly different.
Machine 1 : 
```
Mean : 184.4713592529297  std : 57661.47265625
Mean : 45.32931900024414  std : 57683.6015625
Mean : 26.85606575012207  std : 57691.96875
Mean : 141.6007537841797  std : 57613.0078125
Mean : -53.96173858642578  std : 57649.85546875
Mean : -208.22891235351562  std : 57711.3046875
Mean : -31.99659538269043  std : 57721.828125
Mean : -89.49331665039062  std : 57730.14453125
Mean : 29.143531799316406  std : 57614.54296875
Mean : 35.74909210205078  std : 57695.17578125
```

I am not able to run the gpu version on Machine 2 with the latest torch version because I can't update the nvidia drivers on this computer. I did however get the same behavior with torch 1.5.0 (Slight difference when running Machine 1 vs Machine 2 with GPU).

## Expected behavior
The generated distribution should be identical when generated on every computer.
In an ideal world, the generated distribution should also be identical when generated from CPU and GPU.

## Temporary Solution
To make my model reproducible, I implemented my own initialization method where I replaced `tensor.uniform_()` with the numpy equivalent.
Sample code that show the desired behaviour : 
```python
import torch
import random
import numpy as np

seed = 42

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)

torch.use_deterministic_algorithms(True)

min_val = -99999
max_val = 99999
dim = (146, 520, 4)
for i in range(10):
    tensor = np.random.uniform(min_val, max_val, dim)

    print(f"Mean : {tensor.mean()}  std : {tensor.std()}")
```
Running this on both machine result in the same distributions : 
```
Mean : 95.16237354693021  std : 57729.219329843894
Mean : -44.74602339892699  std : 57673.6178033719
Mean : 190.68677064482338  std : 57759.14380772672
Mean : -123.05053585699686  std : 57677.47312004052
Mean : -156.18468402986596  std : 57679.13498541796
Mean : 17.921712768091773  std : 57734.475463876195
Mean : -55.54245387050414  std : 57729.917024035996
Mean : -90.21646629948282  std : 57765.55274168296
Mean : 13.67219250170903  std : 57727.19657511933
Mean : 169.3708979070732  std : 57737.93892429301
```

## Environment

Machine 1 : 
```
PyTorch version: 1.9.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.25

Python version: 3.6.9 (default, Jan 26 2021, 15:33:00)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-86-generic-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 970
Nvidia driver version: 460.91.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.1
[pip3] torchaudio==0.9.1
[pip3] torchvision==0.10.1
[conda] Could not collect
```

Machine 2 : 
```
PyTorch version: 1.9.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 7.8.2003 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Clang version: Could not collect
CMake version: version 3.6.2
Libc version: glibc-2.3.4

Python version: 3.6.8 (default, Apr  2 2020, 13:34:55)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] (64-bit runtime)
Python platform: Linux-3.10.0-957.21.3.el7.x86_64-x86_64-with-centos-7.8.2003-Core
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: Tesla K40c

Nvidia driver version: 430.26
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.1
[pip3] torchaudio==0.9.1
[pip3] torchvision==0.10.1
[conda] mkl                       2018.0.0             hb491cac_4
[conda] mkl-service               1.1.2            py36h17a0993_4
[conda] numpy                     1.13.3           py36ha12f23b_0
[conda] numpydoc                  0.7.0            py36h18f165f_0
```

## Aditional context
Since the `uniform_()` method is used in the default initializer, this make most model non-reproducible when trained on different machines. In my case, the same code with the same seed resulted in the model getting stuck in a local minima on one machine and achieving > 85% of accuracy on the other.



cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @pbelevich