Repository: pytorch
Issue: Multiprocessing: model shared between processes hangs during copy.deepcopy
Labels: module: multiprocessing,module: cuda,triaged

## 🐛 Bug

I have a model that is used on a couple of processes. Periodically I'd like to be able to duplicate it (and move the new one to a new device) without having to kill the processes using it. Right now, even if the processes aren't actively doing anything, the code will eventually hang during the copy.deepcopy() or during .to(device). What's the proper way to do duplication in this case? 

It doesn't seem to matter for this whether I use share_memory() or not. In my actual use-case I do call share_memory, but it, oddly, wasn't necessary for this repro. Also in practice I'm not sending the same network over and over like this, it just made for a simple repro.

## To Reproduce

Steps to reproduce the behavior:

1. Run the following code sample using PYTHONUNBUFFERED=1 OMP_NUM_THREADS=1 CUDA_VISIBLE_DEVICES=0,1,2,3 repro_script.py --experiment [0, 1, 2, 3]
1. It can be run with fewer devices, as long as the number >=2 
1. 0 is the no-copy case, runs indefinitely. (By the way, if you do deepcopy, but remove one of the put(model_to_share) lines, it also runs indefinitely.)
1. 1 is the copy.deepcopy() case, hangs after 10000-15000 steps on my machine.
2. 2 is the pickle case, hangs after about 5000 steps.
3. 3 is the state_dict case. In this case the state dict seems to load, but the next thing I want to do is move the model to a new device...and that step hangs after about 2 steps.


```python
import torch.multiprocessing as mp
import torch.nn as nn
import torch
import copy
import pickle
import argparse


NUM_DEVICES = torch.cuda.device_count()


def loop_0(q):
    while True:
        q.get()


def loop_1(q):
    while True:
        q.get()


def parse_args():
    parser = argparse.ArgumentParser(description='Run an experiment to demonstrate multiprocess hanging.')
    parser.add_argument('--experiment', type=int, help='0: no-op, 1: copy, 2: pickle, 3: state_dict', required=True)

    args = parser.parse_args()
    return args


def start_processes():
    args = parse_args()
    experiment = args.experiment
    mp.set_start_method('spawn')

    model_to_share = nn.Linear(10, 10)
    model_to_share.cuda()

    loop_0_queue = mp.Queue()
    proc_0 = mp.Process(target=loop_0, args=(loop_0_queue, ))
    proc_0.start()

    loop_1_queue = mp.Queue()
    proc_1 = mp.Process(target=loop_1, args=(loop_1_queue, ))
    proc_1.start()

    success_counter = 0

    while True:
        print("Sending model to processes")
        loop_0_queue.put(model_to_share)
        loop_1_queue.put(model_to_share)
        
        print("Copying")
        # Experiment 0, runs indefinitely
        if experiment == 0:
            print("No-op")
            pass

        # Experiment 1, hangs around 10000-15000:
        elif experiment == 1:
            print("Deep copy")
            model_to_share = copy.deepcopy(model_to_share)

        # Experiment 2, hangs around 5000
        elif experiment == 2:
            print("Pickle")
            model_to_share = pickle.loads(pickle.dumps(model_to_share))

        # Experiment 3, hangs after 2 steps
        elif experiment == 3:
            print("State dict")
            model_to_share.load_state_dict(model_to_share.state_dict())
            print("Moving to device")
            model_to_share.to(f"cuda:{success_counter % NUM_DEVICES}")

        print("Copy complete")

        success_counter += 1
        print(f"Copy counter: {success_counter}.")

if __name__ == "__main__":
    start_processes()

```

## Expected behavior

I would expect all experiment settings [0-3] to run indefinitely without hanging. But really getting just one of them (including the to(device) step after the copy) would be sufficient for my use case.

## Environment

PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: CentOS Linux 7 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
CMake version: version 2.8.12.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 410.78
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.2
[pip] numpydoc==0.9.2
[pip] torch==1.4.0
[pip] torchvision==0.5.0
[conda] _pytorch_select           0.2                       gpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.0.130                      0
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py36he904b0f_0
[conda] mkl_fft                   1.0.15           py36ha843d7b_0
[conda] mkl_random                1.1.0            py36hd6b4f25_0
[conda] numpy                     1.18.2                   pypi_0    pypi
[conda] numpy-base                1.18.1           py36hde5b4d6_1
[conda] numpydoc                  0.9.2                      py_0
[conda] pytorch                   1.4.0           py3.6_cuda10.0.130_cudnn7.6.3_0    pytorch
[conda] torchvision               0.5.0                py36_cu100    pytorch

Since the script says it couldn't collect cuDNN version, here's the output from the conda package list:
cudatoolkit               10.0.130                      0
cudnn                     7.6.5                cuda10.0_0

## Additional context
I believe this is the gdb stack trace from the hanging thread.
```
#0  0x00007ffff7bcd42d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff7bc8dcb in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007ffff7bc8c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fffe2a98667 in torch::(anonymous namespace)::CudaIPCSentDataLimbo::collect() ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#4  0x00007fffe2a9894c in torch::(anonymous namespace)::CudaIPCSentDataDelete(void*) ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#5  0x00007fffe2a964b1 in torch::CudaIPCSentData::~CudaIPCSentData() ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#6  0x00007fffe2a986c5 in torch::(anonymous namespace)::CudaIPCSentDataLimbo::collect() ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#7  0x00007fffe2a98938 in torch::(anonymous namespace)::CudaIPCSentDataDelete(void*) ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#8  0x00007fffb176cc5f in c10::TensorImpl::copy_tensor_metadata(c10::TensorImpl const*, c10::TensorImpl*, c10::VariableVersion const&, bool) ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libc10.so
#9  0x00007fffb177325d in c10::TensorImpl::shallow_copy_from(c10::intrusive_ptr<c10::TensorImpl, c10::detail::intrusive_target_default_null_type<c10::TensorImpl> > const&) () from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libc10.so
#10 0x00007fffb73d4aa2 in torch::autograd::VariableType::set_data(at::Tensor const&, at::Tensor const&) ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch.so
#11 0x00007fffe2beefaa in THPVariable_set_data(THPVariable*, _object*, void*) ()
   from /home/snpowers/miniconda3/envs/venv_pattern_proposal_4/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#12 0x00005555557070ec in _PyObject_GenericSetAttrWithDict () at /tmp/build/80754af9/python_1578429706181/work/Objects/object.c:1189
#13 0x0000555555707597 in PyObject_GenericSetAttr (value=0x7fff94044360, name=<optimized out>, obj=0x7fff9cbf15e8)
    at /tmp/build/80754af9/python_1578429706181/work/Objects/object.c:1231
#14 PyObject_SetAttr () at /tmp/build/80754af9/python_1578429706181/work/Objects/object.c:969
#15 0x000055555570fe53 in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1578429706181/work/Python/ceval.c:2316
```

This issue bears some similarity to https://github.com/pytorch/pytorch/issues/25301, but even if I keep the models around (so they don't go out of scope and get garbage collected), I'm still seeing the hang.


cc @ngimel