Repository: pytorch
Issue: Error with minimal hogwild test (multiprocessing shared memory)
Labels: module: multiprocessing,triaged

## 🐛 Bug

I've encountered a mysterious bug while trying to implement Hogwild with `torch.multiprocessing`. In particular, one version of the code runs fine, but when I add in a seemingly unrelated bit of code before the multiprocessing step, this somehow causes an error during the multiprocessing step: `RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork`

I have reproduced this bug in a toy example below. I am running PyTorch 1.5.1 on Python 3.7.6 on a Linux machine, training on CPU only.

## To Reproduce

The below code produces the error for me. If I comment out the two lines of code `m0 = Model(); train(m0)` which carry out a non-parallel training run on a separate model instance, then everything runs fine. I can't figure out how these lines could be causing a problem.

```python
import torch
import torch.multiprocessing as mp
from torch import nn

def train(model):
    opt = torch.optim.Adam(model.parameters(), lr=1e-5)
    for _ in range(10000):
        opt.zero_grad()
        # We train the model to output the value 4 (arbitrarily)
        loss = (model(0) - 4)**2
        loss.backward()
        opt.step()

# Toy model with one parameter tensor of size 3.
# Output is always the sum of the elements in the tensor,
# independent of the input
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.x = nn.Parameter(torch.ones(3))

    def forward(self, x):
        return torch.sum(self.x)

############################################
# Create a separate Model instance and run
# a non-parallel training run.
# For some reason, this code causes the 
# subsequent parallel run to fail.
m0 = Model()
train(m0)
print ('Done with preliminary run')
############################################

num_processes = 2
model = Model()
model.share_memory()
processes = []
for rank in range(num_processes):
    p = mp.Process(target=train, args=(model,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()
    
print(model.x)
```

Complete output:
```
Done with preliminary run

Process Process-15:
Traceback (most recent call last):
Process Process-16:
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-13-f42ea4d39ecc>", line 11, in train
    loss.backward()
Traceback (most recent call last):
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork
  File "<ipython-input-13-f42ea4d39ecc>", line 11, in train
    loss.backward()
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/spare/local/bwong/.conda/envs/bwong3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork

Parameter containing:
tensor([1., 1., 1.], requires_grad=True)
```

If I comment out the two lines, I get reasonable output:
```
Parameter containing:
tensor([1.1864, 1.1864, 1.1864], requires_grad=True)
```