Repository: pytorch
Issue: Operating on a shared memory tensor with multiple threads hangs
Labels: oncall: distributed,module: multiprocessing,triaged

## 🐛 Bug

Any operation (read or write) on a shared memory tensor hangs if it involves more than `at::GRAIN_SIZE` elements (131076 bytes) because it gets dispatched on multiple threads using `parallel_for`. After calling `ch.set_num_threads(1)` nothing hangs.

## To Reproduce

Steps to reproduce the behavior:

1. Create a tensor in shared memory of more than 128kb
1. Pass it to a sub-process (python or `torch.multiprocessing`)
1. Read or write more than `at::GRAIN_SIZE` in the subprocess
2. It hangs

```python
import torch as ch
from torch.multiprocessing import Process


class MyProcess(Process):

    def __init__(self, shared_data):
        super().__init__()
        self.shared_data = shared_data

    def run(self):
        print("(subproc) first 32k elements", self.shared_data[:-1].sum())
        print("(subproc) last 32k elements", self.shared_data[1:].sum())
        print("(subproc) all elements", self.shared_data.sum())


if __name__ == "__main__":

    data = ch.randn(32769).share_memory_()
    p_bad = MyProcess(data)

    print("(main) first 32k elements", data[:-1].sum())
    print("(main) last 32k elements", data[1:].sum())
    print("(main) all elements", data.sum())

    p_bad.start()
    p_bad.join()
```
Output:

```
(main) first 32k elements tensor(215.6498)
(main) last 32k elements tensor(216.6720)
(main) all elements tensor(216.0387)
(subproc) first 32k elements tensor(215.6498)
(subproc) last 32k elements tensor(216.6720)
```
(and never joins the sub-process)

## Expected behavior

The call returns without hanging

## Environment

From my own bisection it used to work with older version of pytorch and appeared in release `1.5.0`. I don't have resource to recompile pytorch multiple time to find the exact commit though. I assume you already have something like that setup anyway.

Here are the result of the bisection (done with CPU version to make my life easier but they seem to be the same with GPU versions).


<details>
  <summary>First version version exhibiting the bug</summary>

```
PyTorch version: 1.5.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 10.2.0
Clang version: 11.1.0
CMake version: version 3.14.0
Libc version: glibc-2.10

Python version: 3.7 (64-bit runtime)
Python platform: Linux-5.11.16-arch1-1-x86_64-with-arch
Is CUDA available: False
CUDA runtime version: 11.3.58
GPU models and configuration: 
GPU 0: NVIDIA GeForce GT 710
GPU 1: NVIDIA TITAN Xp

Nvidia driver version: 465.24.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torch-tb-profiler==0.1.0
[pip3] torchvision==0.9.1
[conda] _pytorch_select           0.1                       cpu_0  
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] magma-cuda90              2.3.0                         1    pytorch
[conda] mkl                       2019.5                intel_281    intel
[conda] mkl-include               2019.5                intel_281    intel
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] numpy                     1.17.3           py37hd14ec0e_0  
[conda] numpy-base                1.17.3           py37hde5b4d6_0  
[conda] torch                     1.5.0+cpu                pypi_0    pypi
[conda] torchaudio                0.4.0                      py37    pytorch
[conda] torchvision               0.6.0+cpu                pypi_0    pypi
```
</details>
<details>
  <summary>Last version version **not** exhibiting the bug</summary>

```
PyTorch version: 1.4.0
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 10.2.0
Clang version: 11.1.0
CMake version: version 3.14.0
Libc version: glibc-2.10

Python version: 3.7 (64-bit runtime)
Python platform: Linux-5.11.16-arch1-1-x86_64-with-arch
Is CUDA available: False
CUDA runtime version: 11.3.58
GPU models and configuration: 
GPU 0: NVIDIA GeForce GT 710
GPU 1: NVIDIA TITAN Xp

Nvidia driver version: 465.24.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torch-tb-profiler==0.1.0
[pip3] torchvision==0.9.1
[conda] _pytorch_select           0.1                       cpu_0  
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] magma-cuda90              2.3.0                         1    pytorch
[conda] mkl                       2019.5                intel_281    intel
[conda] mkl-include               2019.5                intel_281    intel
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] numpy                     1.17.3           py37hd14ec0e_0  
[conda] numpy-base                1.17.3           py37hde5b4d6_0  
[conda] pytorch                   1.4.0               py3.7_cpu_0  [cpuonly]  pytorch
[conda] torchaudio                0.4.0                      py37    pytorch
[conda] torchvision               0.5.0                  py37_cpu  [cpuonly]  pytorch

```
</details>

## Additional Information

Just it case it might help, here is the stack trace where it hangs:

```
0  0x00007f26c468ec6b in ?? () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libgomp-7c85b1e2.so.1
#1  0x00007f26c468d7a9 in ?? () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libgomp-7c85b1e2.so.1
#2  0x00007f26b528faf4 in at::two_pass_reduction(at::TensorIterator&, c10::function_ref<void (char**, long const*, long, long)>) () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007f26b5290ada in at::TensorIterator::parallel_reduce(c10::function_ref<void (char**, long const*, long, long)>) () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007f26b58b7627 in at::native::(anonymous namespace)::sum_kernel_impl(at::TensorIterator&) () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007f26b51e3958 in void at::native::DispatchStub<void (*)(at::TensorIterator&), at::native::sum_stub>::operator()<at::TensorIterator&>(c10::DeviceType, at::TensorIterator&) () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007f26b51d97eb in at::native::sum_out(at::Tensor&, at::Tensor const&, c10::ArrayRef<long>, bool, c10::optional<c10::ScalarType>) () from /home/guillaume/.conda/envs/pt4/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23