Repository: pytorch
Issue: Issue with dataloader using pin_memory = True
Labels: module: dataloader,triaged

## Issue description
Hello, I'm seeing an odd issue with using the pin_memory = true flag with the dataloader. I'm measuring the time taken to transfer data from the host RAM to GPU memory as follows:

        transfer_time_start = time.time()
        input = input.cuda(args.gpu, non_blocking=False)
        target = target.cuda(args.gpu, non_blocking=False)
        torch.cuda.synchronize()
        transfer_time.update(time.time()-transfer_time_start)

with pin_memory = True in the dataloader, this gives me a transfer time of 0.03 sec, which for a batch size of 256, translates into 256 times 224 times 224 times 3 times 4/0.03 = 5.1GB, which is a bit low for my CPU-GPU interconnect (x16, PCIe3) which should deliver ~12GB. 

I then tried calling pin_memory() manually on the tensor returned by the enumerate call, as shown below:

for i, (input, target) in enumerate(train_loader):
        input = input.pin_memory()
        # measure data loading time
        data_time.update(time.time() - end)
        transfer_time_start = time.time()
        input = input.cuda(args.gpu, non_blocking=False)
        target = target.cuda(args.gpu, non_blocking=False)
        torch.cuda.synchronize()
        transfer_time.update(time.time()-transfer_time_start)

Now the transfer time dropped to 0.014, which translates to ~11GB, which is as expected. Anyone has any ideas why setting pin_memory = True in the data loader may not return a tensor already in pinned memory?

Also attached below are two plots showing the transfer time (green plot) from host memory to the GPU.
This plot shows the transfer time when I call pin_memory manually
![image](https://user-images.githubusercontent.com/6232547/52499150-c3bdb280-2ba8-11e9-92ec-d2556079f95e.png)

You can see that the transfer time stays consistently low.

Whereas this one shows the transfer time without calling pin_memory manually. Now the transfer time is highly variable and averages to around 0.03 sec
![image](https://user-images.githubusercontent.com/6232547/52499161-cb7d5700-2ba8-11e9-872b-284c2c678100.png)

If I insert a sleep of 50 ms after the enumerate call, I again obtain a nice, low transfer time. This indicates that the data loader simply needs a bit more time to finish the pinned_memory transfer? But inserting the sleep shouldn't be necessary as the dataloader shouldn't return from the enumerate call if the transfer is not complete yet? 

## System Info
Please copy and paste the output from our
PyTorch version: 1.0.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.1.85
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 387.26
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.2

Versions of relevant libraries:
[pip] numpy==1.15.4
[pip] torch==1.0.0
[pip] torchvision==0.2.1
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.1                      144  
[conda] mkl_fft                   1.0.10           py37ha843d7b_0  
[conda] mkl_random                1.0.2            py37hd81dba3_0  
[conda] pytorch                   1.0.0           py3.7_cuda9.0.176_cudnn7.4.1_1    pytorch
[conda] torchvision               0.2.1                      py_2    pytorch


cc @SsnL