Repository: pytorch
Issue: MultiGPU hangs Titan Xp in multiprocessing/queue.py
Labels: module: multiprocessing,module: cuda,triaged,module: macos

PyTorch GitHub Issues Guidelines
--------------------------------

We like to limit our issues to bug reports and feature requests. If you have a question or would like help and support, please visit our forums: https://discuss.pytorch.org/

If you are submitting a feature request, please preface the title with [feature request].

When submitting a bug report, please include the following information (where relevant):
- OS:
- PyTorch version:
- How you installed PyTorch (conda, pip, source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- GCC version (if compiling from source):

In addition, including the following information will also be very helpful for us to diagnose the problem:
- A script to reproduce the bug. Please try to provide as minimal of a test case as possible.
- Error messages and/or stack traces of the bug
- Context around what you are trying to do


**Hi there,**

I have been trying for about a week and a half to get all 3 of my NVIDIA Titan Xp, but the process always hangs in the multiprocessing directory in the "queues.py" and the "synchronizing.py" whenever I use more than 1 of them.

MY SETUP:
- OS: macOS Sierra 10.12.5
- PyTorch version: 0.3.0.post4
- PyTorch installation via pip
-Python version: 3.5
-CUDA/cuDNN version: CUDA 8.0.61  and torch.backends.cudnn.version() = 7003
-GPU models and configuration: 3 NVIDIA Titan Xps installed on PCI express board 


This problem seems to occur whenever I use the data parallel functionality in PyTorch. It happens on very simple calculations and in things like ConvNets etc. It loads the network architecture and then gets stuck at what seems to be the beginning of training the network.

> Traceback (most recent call last):
  File "train.py", line 229, in <module>
    output = netD(inputv)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/chris/Documents/MPhilProjects/GANs/PyTorchApproach/DCGAN-pytorch/model.py", line 110, in forward
    output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 113, in data_parallel
    outputs = parallel_apply(replicas, inputs, module_kwargs, used_device_ids)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 59, in parallel_apply
    thread.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    r = index_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 343, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
Process Process-2:
    elif lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    r = index_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 342, in get
    with self._rlock:
  File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 96, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process Process-5:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    r = index_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 342, in get
    with self._rlock:
  File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 96, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 1288, in _shutdown
    t.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

I tried looking at the connections b/w them and got the following result:
![image](https://user-images.githubusercontent.com/6062598/35688012-1823ef76-0768-11e8-8fca-c6147df47a80.png)

I tried testing the p2pBandwidth example that comes in cuda and got the following result:
![image](https://user-images.githubusercontent.com/6062598/35688187-786df2aa-0768-11e8-87d0-a6c7462df3bd.png)


But I am still essentially clueless as to why they won't work together. Any advice / solutions would be **greatly** appreciated!

Best,
Michael



cc @ngimel