Repository: pytorch
Issue: Pytorch streams API don't execute concurrently, However Same code in CUDA does. 
Labels: module: performance,module: cuda,triaged

## 🐛 Bug

Pytorch streams don't execute in parallel. 

### To Reproduce

Steps to reproduce the behavior (**always include the command you ran**):

1. Run following script:
```py
import torch
input = []
batch_size = 64
shape = (batch_size, 64, 256)
for i in range (0, 4):
    input.append(batch)
input = [in_.pin_memory() for in_ in input]

s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()
s3 = torch.cuda.Stream()
s4 = torch.cuda.Stream()

torch.cuda.synchronize()
cpu_device = torch.device('cpu')
device = torch.device('cuda:0')

with torch.cuda.stream(s1):
    curr_batch = input[0].to(device,non_blocking = True)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    input[0].copy_(curr_batch,non_blocking = True)

with torch.cuda.stream(s2):
    curr_batch = input[1].to(device,non_blocking = True)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    input[1].copy_(curr_batch,non_blocking = True)


with torch.cuda.stream(s3):
    curr_batch = input[2].to(device,non_blocking = True)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    input[2].copy_(curr_batch,non_blocking = True)


with torch.cuda.stream(s4):
    curr_batch = input[3].to(device,non_blocking = True)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    curr_batch = torch.exp (curr_batch)
    input[3].copy_(curr_batch,non_blocking = True)


torch.cuda.synchronize()
```

Nvidia Systems Profile attached. 
![image](https://user-images.githubusercontent.com/66033489/99738086-90e36680-2a7e-11eb-8f1c-612038af1a7a.png)

**Similar code converted to CUDA.** 
https://github.com/NVIDIA-developer-blog/code-samples/blob/master/series/cuda-cpp/overlap-data-transfers/async.cu

Profile: 
![image](https://user-images.githubusercontent.com/66033489/99739568-e5d4ac00-2a81-11eb-94c8-b1ac83a4bc10.png)


### Expected behavior

Ideally behavior should be same as CUDA code. As can be clearly seen in Nsight systems profile, Pytorch streams API do not seem to work. I also tried modifying data input size to eliminate the possibility of kernel launching overhead to be large. 

## Environment
```
Collecting environment information...
PyTorch version: 1.5.0a0+8f84ded
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.14.0

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB
GPU 2: Tesla V100-PCIE-16GB
GPU 3: Tesla V100-PCIE-16GB

Nvidia driver version: 440.33.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] msgpack-numpy==0.4.3.2
[pip3] numpy==1.18.1
[pip3] pytorch-transformers==1.1.0
[pip3] torch==1.5.0a0+8f84ded
[pip3] torchtext==0.4.0
[pip3] torchvision==0.6.0a0
[conda] magma-cuda101             2.5.1                         1    local
[conda] mkl                       2019.1                      144
[conda] mkl-include               2019.1                      144
[conda] msgpack-numpy             0.4.3.2                  py36_0
[conda] nomkl                     3.0                           0
[conda] numpy                     1.18.1           py36h94c655d_0
[conda] numpy-base                1.18.1           py36h2f8d375_1
[conda] pytorch-transformers      1.1.0                    pypi_0    pypi
[conda] torch                     1.5.0a0+8f84ded          pypi_0    pypi
[conda] torchtext                 0.4.0                    pypi_0    pypi
[conda] torchvision               0.6.0a0                  pypi_0    pypi
```

cc @ngimel @VitalyFedyunin