Repository: pytorch
Issue: non-shuffling data loaders can affect random states, thus the results of shuffling data loaders.
Labels: todo,module: dataloader,triaged

## Issue description

See the title. While it's arguable this behavior is bug or not, I feel it's not very intuitive. Maybe it's good to doc this.

For an actual example, see below.

In the example, I have two data loaders, one shuffling for training, one non-shuffling for validation. Turns out, the order of data sample points for the shuffling training loader depends on how many times I call the non-shuffling validation loader first.

Overall, I think making **behaviorally deterministic** operations affect random states is certainly unintuitive; for example, you would not expect that running a few matrix multiplications in `numpy` can affect a random number generation operation that follows these matrix multiplications.

## Code example

```python
# In[1]:


from torch.utils.data import DataLoader
from torch.utils.data.dataset import TensorDataset
import torch
print(torch.__version__)


# In[2]:


import numpy as np
np.random.seed(0)

# In[3]:


dataset_train = np.random.randint(100, size=(10,)).astype(np.float32)
dataset_val = np.random.randint(100, size=(10,)).astype(np.float32)
print('train\n',dataset_train)
print('val\n',dataset_val)

# train
# [44. 47. 64. 67. 67.  9. 83. 21. 36. 87.]
# val
# [70. 88. 88. 12. 58. 65. 39. 87. 46. 88.]

# In[4]:


dataloader_train = DataLoader(TensorDataset(torch.tensor(dataset_train)),
                                            batch_size=10, shuffle=True)
dataloader_val = DataLoader(TensorDataset(torch.tensor(dataset_val)),
                         batch_size=10, shuffle=False)
def loop_one(dataloader_this):
    for (x,) in dataloader_this:
        print(x)




# In[5]:


# only train
np.random.seed(0)
torch.manual_seed(0)
loop_one(dataloader_train)

# no val, we get the following for dataloader_train
# tensor([67., 21.,  9., 64., 44., 36., 47., 83., 87., 67.])

# In[6]:


# val + train
np.random.seed(0)
torch.manual_seed(0)
loop_one(dataloader_val)
loop_one(dataloader_train)

# one val
# tensor([67., 21.,  9., 83., 64., 36., 47., 87., 67., 44.])

# In[7]:


# val + val + train
np.random.seed(0)
torch.manual_seed(0)
loop_one(dataloader_val)
loop_one(dataloader_val)
loop_one(dataloader_train)

# two val
# tensor([21., 64., 67., 36., 87., 83.,  9., 44., 47., 67.])

# In[8]:


# val + val + reseed +  train
np.random.seed(0)
torch.manual_seed(0)
loop_one(dataloader_val)
loop_one(dataloader_val)
np.random.seed(0)
torch.manual_seed(0)
loop_one(dataloader_train)

# reseed
# tensor([67., 21.,  9., 64., 44., 36., 47., 83., 87., 67.])
```


## System Info

I ran that `collect_env.py` script  under a Singularity container with Ubuntu 14.04 host.

```
Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX TITAN Black
GPU 1: Tesla K40c

Nvidia driver version: 384.111
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```


cc @SsnL @VitalyFedyunin @ejguan