Repository: pytorch
Issue: [threaded pg] All threads share one Random Number Generator
Labels: triaged,module: random,module: dtensor

### 🐛 Describe the bug

In `MultiThreadedTestCase`'s worker threads, `torch.rand()` shares one Random Number Generator which hurts the assumption in some tests that `torch.rand()` after `torch.manual_seed(CONST)` will give the same result within each worker. 

To verify:
```
# add this test
class ReproduceRNG(MultiThreadedTestCase):
        def test_random_seed_consistency(self):
                self_tensor = torch.rand(3, 3)
                print(f"from rank {dist.get_rank()}")
                print(self_tensor)

# modify MultiThreadedTestCase
class MultiThreadedTestCase(TestCase):
        ...

        def perThreadSetUp(self):
                super().setUp()
                torch.manual_seed(5)
                print("from master thread:")
                for i in range(4):
                    print(torch.rand(3, 3))
                torch.manual_seed(5)
```

Output:
```
from master thread:
tensor([[0.8303, 0.1261, 0.9075],
        [0.8199, 0.9201, 0.1166],
        [0.1644, 0.7379, 0.0333]])
tensor([[0.9942, 0.6064, 0.5646],
        [0.0724, 0.6593, 0.7150],
        [0.5793, 0.9809, 0.6502]])
tensor([[0.0566, 0.9201, 0.6698],
        [0.2615, 0.0407, 0.7850],
        [0.9752, 0.0903, 0.5273]])
tensor([[0.6794, 0.2639, 0.3906],
        [0.1661, 0.2636, 0.0442],
        [0.4884, 0.7965, 0.7432]])
from rank 3
from rank 0
from rank 1
from rank 2
tensor([[0.8303, 0.1261, 0.9075],
        [0.8199, 0.9201, 0.1166],
        [0.1644, 0.7379, 0.0333]])
tensor([[0.9942, 0.6064, 0.5646],
        [0.0724, 0.6593, 0.7150],
        [0.5793, 0.9809, 0.6502]])
tensor([[0.6794, 0.2639, 0.3906],
        [0.1661, 0.2636, 0.0442],
        [0.4884, 0.7965, 0.7432]])
tensor([[0.0566, 0.9201, 0.6698],
        [0.2615, 0.0407, 0.7850],
        [0.9752, 0.0903, 0.5273]])
```

### Versions

Collecting environment information...
PyTorch version: 1.14.0a0+git3dda91e
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-1069-aws-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.6.112
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.4
[pip3] torch==1.14.0a0+git3dda91e
[conda] blas                      1.0                         mkl  
[conda] magma-cuda116             2.6.1                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-include               2022.1.0           h06a4308_224  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] numpy-base                1.23.4          py310h8e6c178_0  
[conda] torch                     1.14.0a0+git3dda91e           dev_0    <develop>


cc @pbelevich