Repository: pytorch
Issue: Operating on CUDA tensors on multiprocessing queue hangs
Labels: module: windows,module: multiprocessing,triaged

# 🐛 Bug

Reading Cuda tensors from multiprocessing queue causes child (reader) process to hang.
I discovered that process hangs on any operation (sum, min, max, mean, etc.), in my example code on "imgs.mean()".
This code hangs on Windows machine, but works fine on *unix (ununtu-22.10).


## To Reproduce

```
import torch
import queue
import torch.multiprocessing as mp

class MyProcess(mp.Process):
    def __init__(self):
        super().__init__()
        self.queue = mp.Queue(200)

    def run(self):
        while 1:
            print('running')
            try:
                imgs = self.queue.get(timeout=1)
                print("queue.get() ended, got", imgs.mean())

                del imgs # save memory
            except queue.Empty:
                print("queue.Empty")
                continue

    def write_frames(self, a):
        self.queue.put(a)

if __name__ == "__main__":
    mp.set_start_method('spawn')
    torch.set_num_threads(1)

    proc = MyProcess()
    for i in range(10):
        a = torch.rand([8,1,320,320], device='cuda') + i
        proc.write_frames(a)
        del a # save memory
    proc.start()
    proc.join()
```

## Output:
```
running
queue.get() ended, got tensor(0., device='cuda:0')
running
queue.get() ended, got tensor(0., device='cuda:0')
running
queue.get() ended, got tensor(2.5001, device='cuda:0')
running
queue.get() ended, got tensor(3.4997, device='cuda:0')
running
queue.get() ended, got tensor(4.4997, device='cuda:0')
running
queue.get() ended, got 
```
(and never joins the sub-process)

## Expected behavior
The join call returns without hanging

### Versions
[pip3] torch==2.1.1

<details>
<summary>Environment full</summary>

Collecting environment information...
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.24.0
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.128
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU
Nvidia driver version: 546.33
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             16
On-line CPU(s) list:                0-15
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen 9 6900HX with Radeon Graphics
CPU family:                         25
Model:                              68
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           6587.44
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm
Virtualization:                     AMD-V
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          256 KiB (8 instances)
L1i cache:                          256 KiB (8 instances)
L2 cache:                           4 MiB (8 instances)
L3 cache:                           16 MiB (1 instance)
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] onnx==1.15.0
[pip3] onnxruntime==1.16.3
[pip3] onnxsim==0.4.35
[pip3] torch==2.1.2
[pip3] torchvision==0.16.1
[pip3] triton==2.1.0
[conda] Could not collect

</details>


cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @VitalyFedyunin