Repository: pytorch
Issue: Dataloader's memory usage keeps increasing during one single epoch.
Labels: module: dataloader,triaged

(All codes were tested on Pytorch 1.0.0 and Pytorch 1.0.1. Memory capacity of my machine is `256Gb`. )

## Description and Reproduction

Hi,

I create a dataloader to load features from local files by their file paths but find this results in OOM problem even though the code is simple.

The dataloader can be simplfied as:

```python
import numpy as np
import torch
import torch.utils.data as data
import time

class MyDataSet(data.Dataset):

  def __init__(self):
    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset,
    # such as a list of file names or paths.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset is 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):
    info = self.infoset[index]  # problem is here
    items = {}
    items['features'] = self.load_feature(info)
    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))
    return feature

  def __len__(self):
    return 8000000


dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:
  for i, sample in enumerate(dataloader):
    print(i, len(dataloader))
    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch
```

During each epoch, the memory usage is about `13GB` at the very beginning and keeps inscreasing and finally up to about `46Gb`, like this:

![Annotation 2019-05-13 191331](https://user-images.githubusercontent.com/21999660/57630625-89f9af00-75d0-11e9-830b-c8e6ede5b322.png)


Although it will decrease to `13GB` at the beginning of next epoch, this problem is serious to me because in my real project the `infoset` is about `40Gb` due to the large number of samples and finally leads to `Out of Memory` (OOM) at the end of the first epoch.

## Expected behavior

I have found that the problem is caused by the first line of `MyDataset.__getitem__()` : `info = self.infoset[index]`, in the following code, if I remove this line, then memory usage is normal, which is also my expected behavior.

```python
class MyDataSet(data.Dataset):

  def __init__(self):

    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset may be 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):

    # info = self.infoset[index]  # problem is here
    info = 'fake info'

    items = {}
    items['features'] = self.load_feature(info)

    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))

    return feature

  def __len__(self):

    return 8000000

dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:

  for i, sample in enumerate(dataloader):

    print(i, len(dataloader))

    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch
```

And the corresponding mem usage is stable at `13GB`:

![Annotation 2019-05-13 191509](https://user-images.githubusercontent.com/21999660/57631324-dd203180-75d1-11e9-8386-ea3be4609543.png)

## More test

In the following code, I don't even load features in `__getitem__()` but just read a string of `infoset`, but get the same problem:

```python
class MyDataSet(data.Dataset):

  def __init__(self):

    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset may be 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):

    info = self.infoset[index]  # problem is here

    items = {}
    # items['features'] = self.load_feature(info)

    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))

    return feature

  def __len__(self):

    return 8000000

dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:

  for i, sample in enumerate(dataloader):

    print(i, len(dataloader))

    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch

```

Mem usage:


![Annotation 2019-05-13 191418](https://user-images.githubusercontent.com/21999660/57631896-f37abd00-75d2-11e9-90f5-9139eb4215b8.png)

Any suggestions or reasons about this problem?

Thanks.