Repository: pytorch
Issue: "Attempted to send CUDA tensor received from another process" even though that was not the case
Labels: module: multiprocessing,triaged

### 🐛 Describe the bug

Consider the simple example, written by following all the guidelines regarding multiprocessing and torch:

```python
import torch.multiprocessing as mp
import logging
from logging.handlers import QueueHandler, QueueListener
import traceback

import torch

def image_writer(queue, i, logging_queue):
    logger = logging.getLogger("foobar")
    logger.setLevel(logging.INFO)
    handler = QueueHandler(logging_queue)
    handler.setLevel(logging.INFO)
    logger.addHandler(handler)
    n = 0

    try:
        logger.info(f"[{i}] Starting write loop")
        while True:
            item = queue.get()
            if item is None:
                logger.info(f"[{i}] Exiting write loop")
                break
            logger.info(f"[{i}] got item {item}")
            n += 1
    except Exception:
        logger.error(f"[{i}] {traceback.format_exc()}")
        raise
    finally:
        logger.debug(f"[{i}] Exiting write loop")

    return i, n


if __name__ == '__main__':
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.INFO)

    h2 = logging.StreamHandler()
    f2 = logging.Formatter('MAIN: %(message)s')
    h2.setFormatter(f2)
    logger.addHandler(h2)

    logger.info("Start")

    manager = mp.get_context('spawn').Manager()

    _logging_queue = manager.Queue()
    _logging_listener = QueueListener(_logging_queue, *logger.handlers)
    _logging_listener.start()

    queue = manager.Queue()
    _n_writers = 2
    pool = manager.Pool(_n_writers)
    pool_results = []
    for i in range(_n_writers):
        pool_results.append(pool.apply_async(image_writer, (queue, i, _logging_queue)))

    for i in range(10):
        t = torch.tensor([42+i]).cuda()
        # clone = t.cpu().share_memory_() # This works
        clone = t.clone() # This doesn't
        queue.put(clone)

    for i in range(_n_writers):
        queue.put(None)

    for r in pool_results:
        i, n = r.get(10)
        logger.info(f"Process {i} Received {n} items")

    logger.info("Stopping")
    _logging_listener.stop()
    manager.shutdown()
```

Running it I get something like this:
```shell
MAIN: Start
MAIN: [0] Starting write loop
MAIN: [1] Starting write loop
MAIN: [0] Traceback (most recent call last):
  File "/tmp/test.py", line 19, in image_writer
    item = queue.get()
           ^^^^^^^^^^^
  File "<string>", line 2, in get
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 836, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 308, in serve_client
    send(msg)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_tensor
    ) = storage._share_cuda_()
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/storage.py", line 1200, in _share_cuda_
    return self._untyped_storage._share_cuda_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending.

---------------------------------------------------------------------------

Traceback (most recent call last):
  File "/tmp/test.py", line 68, in <module>
    i, n = r.get(10)
           ^^^^^^^^^
  File "<string>", line 2, in get
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 836, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 308, in serve_client
    send(msg)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_tensor
    ) = storage._share_cuda_()
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/storage.py", line 1200, in _share_cuda_
    return self._untyped_storage._share_cuda_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending.

---------------------------------------------------------------------------
MAIN: [1] Traceback (most recent call last):
  File "/tmp/test.py", line 19, in image_writer
    item = queue.get()
           ^^^^^^^^^^^
  File "<string>", line 2, in get
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 836, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 308, in serve_client
    send(msg)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_tensor
    ) = storage._share_cuda_()
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/site-packages/torch/storage.py", line 1200, in _share_cuda_
    return self._untyped_storage._share_cuda_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempted to send CUDA tensor received from another process; this is not currently supported. Consider cloning before sending.

---------------------------------------------------------------------------

Exception in thread Thread-1 (_monitor):
Traceback (most recent call last):
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/logging/handlers.py", line 1574, in _monitor
    record = self.dequeue(True)
             ^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/logging/handlers.py", line 1523, in dequeue
    return self.queue.get(block)
           ^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in get
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/managers.py", line 821, in _callmethod
    kind, result = conn.recv()
                   ^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/home/paperspace/.conda/envs/EDVR/lib/python3.12/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
[W828 23:57:56.977435534 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
```

All the tensors I am putting in the queue are clearly not from another process, and should not even require cloning. But even forcing a clone as the error message requests does not resolve the problem. What am I doing wrong there?


### Versions

Collecting environment information...             
PyTorch version: 2.4.0+cu121                                                                              
Is debug build: False                             
CUDA used to build PyTorch: 12.1                  
ROCM used to build PyTorch: N/A                   
                                                                                                          
OS: Ubuntu 20.04.4 LTS (x86_64)                                                                                                                                                                                     
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0                                                                                                                                                                  
Clang version: Could not collect                  
CMake version: version 3.16.3                     
Libc version: glibc-2.31                                                                                                                                                                                                                                                                                                                                                                                                                Python version: 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:23:07) [GCC 12.3.0] (64-bit runtime)                                                                                                      Python platform: Linux-5.15.0-117-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.124 
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A  
MIOpen runtime version: N/A
Is XNNPACK available: True
                                                                                                          
CPU:                                                                                                      
Architecture:                         x86_64                                                              
CPU op-mode(s):                       32-bit, 64-bit                                                      
Byte Order:                           Little Endian                                                       
Address sizes:                        46 bits physical, 48 bits virtual
CPU(s):                               12
On-line CPU(s) list:                  0-11
Thread(s) per core:                   1                                                                                                                                                                             
Core(s) per socket:                   12
Socket(s):                            1           
NUMA node(s):                         1                                                                                                                                                                             
Vendor ID:                            GenuineIntel                                                                                                                                                                  
CPU family:                           6                                                                   
Model:                                106
Model name:                           Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz                                                                                                                                      
Stepping:                             6                                                                                                                                                                             
CPU MHz:                              2800.210                                                            
BogoMIPS:                             5600.76 
Hypervisor vendor:                    Xen    
Virtualization type:                  full   
L1d cache:                            576 KiB        
L1i cache:                            384 KiB                                                                                                                                                                       
L2 cache:                             15 MiB         
L3 cache:                             432 MiB                                                             
NUMA node0 CPU(s):                    0-11           
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected   
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected                                                        
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected                                                        
Vulnerability Spec rstack overflow:   Not affected                                                                                                                                                                  
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp                                                                                                           
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization 
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected                                                                                                                                                                  Vulnerability Tsx async abort:        Not affected                                                                                                                                                                  Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single intel_ppin ssbd ibrs ib
pb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves umip pku ospke gfni vaes vpclmulqdq rdpid md_clear flush_l
1d arch_capabilities           
                                                     
Versions of relevant libraries:                                                                           
[pip3] mypy-extensions==1.0.0    
[pip3] numpy==2.0.1             
[pip3] onnx==1.16.1       
[pip3] torch==2.4.0        
[pip3] torchmetrics==1.4.1
[pip3] torchvision==0.19.0                                                                                
[pip3] triton==3.0.0                                                                                      
[conda] numpy                     2.0.1                    pypi_0    pypi                                 
[conda] torch                     2.4.0                    pypi_0    pypi                                 
[conda] torchmetrics              1.4.1                    pypi_0    pypi                                 
[conda] torchvision               0.19.0                   pypi_0    pypi                                 
[conda] triton                    3.0.0                    pypi_0    pypi


cc @VitalyFedyunin