Repository: pytorch
Issue: thread blocked when moving a tensor from GPU to CPU, by calling the function .cpu() in pytorch. This kind of block can be stop by any window event like mouse moving/clicking or keyboard pressing.
Labels: module: cuda,module: cpu,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
The thread blocked or the process was hanging when moving a tensor's memory access from GPU to CPU, by calling the function .cpu() or to('cpu') in pytorch. This kind of block can be stop by any window event like mouse moving/clicking or keyboard pressing, and even a pop-up system notification.

## To Reproduce

Steps to reproduce the behavior:

1. Run the sample code and check the print. If the print runs smoothly, that means the issue has not reproduced, please increase the "tensorSize" in code. If the print stops frequently, and the "get out cost" in log is not stable, that means the issue reproduced.
2. Run the shell script in a new terminal, that's to simulate window events in a period of 0.01s.
3. Check the print, it should be runs smoothly, and the "get out cost" in log is stable.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
### sample code
```python
import time
import torch

if __name__ == '__main__':
    tartTime = time.time()
    tensorSize = 224
    with torch.no_grad():
        while (True):
            timeBeforePutIn = time.time()
            pred = torch.ones([1, tensorSize, tensorSize, tensorSize], dtype=torch.float32, device='cuda')
            timeAfterPutIn = time.time()

            torch.cuda.synchronize()
            timeAfterSync = time.time()

            predOnCpu = pred.detach().cpu()
            timeAfterGetOut = time.time()

            print(" put in cost: {:.2f} ms,".format((timeAfterPutIn - timeBeforePutIn) * 1000),
                  " sync cost: {:.2f} ms,".format((timeAfterSync - timeAfterPutIn) * 1000),
                  " get out cost: {:.2f} ms".format((timeAfterGetOut - timeAfterSync) * 1000))
```
### shell script to creating a window event every 0.01s
```shell
#!/bin/bash
int=1
while(( $int==1 ))
do
    echo
    sleep 0.01
done
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The "get out cost" in this issue is the time cost of code "pred.detach().cpu()", it moves the a tensor from GPU to CPU. It is expected to be finished in a short time, like the following log shows. 

But in fact, it is different when there is window event happens or not. In this case, the window event includes: the mouse swiping on the window, typing in the terminal, a pop-up system notification, etc. 

I simulated a window event in the above shell script, that's typing a empty space in terminal every 0.01s. When the script is running, window event occurs continuously, the program runs well, the "get out cost" time not exceed 100 ms, the log as follows:
```shell
 put in cost: 0.06 ms,  sync cost: 17.12 ms,  get out cost: 84.01 ms
 put in cost: 0.06 ms,  sync cost: 19.59 ms,  get out cost: 66.65 ms
 put in cost: 0.05 ms,  sync cost: 17.18 ms,  get out cost: 90.63 ms
 put in cost: 0.05 ms,  sync cost: 16.86 ms,  get out cost: 84.06 ms
 put in cost: 0.06 ms,  sync cost: 16.99 ms,  get out cost: 95.12 ms
```

But when there is no window events to trigger, the thread may blocks at "pred.detach().cpu()" in code, and the "get out cost" time may reaching **several seconds** and even one minute. The log as follows:
```shell
 put in cost: 0.06 ms,  sync cost: 3.13 ms,  get out cost: 47.16 ms
 put in cost: 0.06 ms,  sync cost: 16.77 ms,  get out cost: 48.81 ms
 put in cost: 0.07 ms,  sync cost: 3.23 ms,  get out cost: 44.78 ms
 put in cost: 0.06 ms,  sync cost: 4.64 ms,  get out cost: 33.21 ms
 put in cost: 0.06 ms,  sync cost: 7.93 ms,  get out cost: 2014.91 ms
 put in cost: 0.08 ms,  sync cost: 387.73 ms,  get out cost: 835.83 ms
 put in cost: 0.08 ms,  sync cost: 149.06 ms,  get out cost: 857.78 ms
 put in cost: 0.06 ms,  sync cost: 83.32 ms,  get out cost: 978.19 ms
 put in cost: 0.07 ms,  sync cost: 110.55 ms,  get out cost: 1500.69 ms
 put in cost: 0.07 ms,  sync cost: 115.90 ms,  get out cost: 2206.20 ms
 put in cost: 0.07 ms,  sync cost: 184.32 ms,  get out cost: 991.34 ms
 put in cost: 0.06 ms,  sync cost: 1200.28 ms,  get out cost: **6001.93 ms**
 put in cost: 0.08 ms,  sync cost: 128.42 ms,  get out cost: 4672.73 ms
```

## Environment

root@6ffacbcb6331:~# python collect_env.py 
```shell
Collecting environment information...
PyTorch version: 1.7.0a0+fc1103a
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 3.8.20371-d1886b0b

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.6.3

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: Device 1636
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 3.18.6
MIOpen runtime version: 2.6.0

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.7.0a0+fc1103a
[pip3] torchvision==0.8.0a0+ac3ba94
[conda] Could not collect
```
### Supplement

 - Platform information:
CPU: AMD Ryzen 4500U
GPU: AMD Vega 6
OS: Ubuntu18.04
ROCM: rocm-3.8.0

 - How you installed PyTorch (`conda`, `pip`, source):
Install the pytorch in a docker, based on ROCm platform.
The installing command as follows:
```shell
sudo docker pull rocm/pytorch:rocm3.8_ubuntu18.04_py3.6_pytorch
```

 - Any other relevant information:
```shell
root@6ffacbcb6331:~/fast-depth# clinfo
Number of platforms:                             1
  Platform Profile:                              FULL_PROFILE
  Platform Version:                              OpenCL 2.0 AMD-APP (3186.0)
  Platform Name:                                 AMD Accelerated Parallel Processing
  Platform Vendor:                               Advanced Micro Devices, Inc.
  Platform Extensions:                           cl_khr_icd cl_amd_event_callback 


  Platform Name:                                 AMD Accelerated Parallel Processing
Number of devices:                               1
  Device Type:                                   CL_DEVICE_TYPE_GPU
  Vendor ID:                                     1002h
  Board name:                                    Device 1636
  Device Topology:                               PCI[ B#4, D#0, F#0 ]
  Max compute units:                             26
  Max work items dimensions:                     3
    Max work items[0]:                           1024
    Max work items[1]:                           1024
    Max work items[2]:                           1024
  Max work group size:                           256
  Preferred vector width char:                   4
  Preferred vector width short:                  2
  Preferred vector width int:                    1
  Preferred vector width long:                   1
  Preferred vector width float:                  1
  Preferred vector width double:                 1
  Native vector width char:                      4
  Native vector width short:                     2
  Native vector width int:                       1
  Native vector width long:                      1
  Native vector width float:                     1
  Native vector width double:                    1
  Max clock frequency:                           1500Mhz
  Address bits:                                  64
  Max memory allocation:                         456340275
  Image support:                                 Yes
  Max number of images read arguments:           128
  Max number of images write arguments:          8
  Max image 2D width:                            16384
  Max image 2D height:                           16384
  Max image 3D width:                            2048
  Max image 3D height:                           2048
  Max image 3D depth:                            2048
  Max samplers within kernel:                    5686
  Max size of kernel argument:                   1024
  Alignment (bits) of base address:              1024
  Minimum alignment (bytes) for any datatype:    128
  Single precision floating point capability
    Denorms:                                     Yes
    Quiet NaNs:                                  Yes
    Round to nearest even:                       Yes
    Round to zero:                               Yes
    Round to +ve and infinity:                   Yes
    IEEE754-2008 fused multiply-add:             Yes
  Cache type:                                    Read/Write
  Cache line size:                               64
  Cache size:                                    16384
  Global memory size:                            536870912
  Constant buffer size:                          456340275
  Max number of constant args:                   8
  Local memory type:                             Scratchpad
  Local memory size:                             65536
  Max pipe arguments:                            16
  Max pipe active reservations:                  16
  Max pipe packet size:                          456340275
  Max global variable size:                      456340275
  Max global variable preferred total size:      536870912
  Max read/write image args:                     64
  Max on device events:                          1024
  Queue on device max size:                      8388608
  Max on device queues:                          1
  Queue on device preferred size:                262144
  SVM capabilities:                              
    Coarse grain buffer:                         Yes
    Fine grain buffer:                           Yes
    Fine grain system:                           No
    Atomics:                                     No
  Preferred platform atomic alignment:           0
  Preferred global atomic alignment:             0
  Preferred local atomic alignment:              0
  Kernel Preferred work group size multiple:     64
  Error correction support:                      0
  Unified memory for Host and Device:            0
  Profiling timer resolution:                    1
  Device endianess:                              Little
  Available:                                     Yes
  Compiler available:                            Yes
  Execution capabilities:                                
    Execute OpenCL kernels:                      Yes
    Execute native function:                     No
  Queue on Host properties:                              
    Out-of-Order:                                No
    Profiling :                                  Yes
  Queue on Device properties:                            
    Out-of-Order:                                Yes
    Profiling :                                  Yes
  Platform ID:                                   0x7f8a50aa8cd0
  Name:                                          gfx900
  Vendor:                                        Advanced Micro Devices, Inc.
  Device OpenCL C version:                       OpenCL C 2.0 
  Driver version:                                3186.0 (HSA1.1,LC)
  Profile:                                       FULL_PROFILE
  Version:                                       OpenCL 2.0 
  Extensions:                                    cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program 
```

## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin @ngimel