Repository: pytorch
Issue: PyTorch 1.4.0 CUDA initialization error with CPU-only (multiprocessing) on Python 3.7.5
Labels: module: autograd,module: cuda,triaged

## Bug
Using only the CPU device, PyTorch 1.4.0 crashes on Python 3.7.5 when training a neural network on the main process and then training a different network on child processes.

## To Reproduce

Steps to reproduce the behavior:

1. Train a neural network on the parent process
2. Create 5 child processes and train the neural network on them, sleeping to wait for background activity
3. Observe the error

**Code to Reproduce**
```import random
from multiprocessing import Process, Queue
import time

import torch
import torch.nn as nn
import torch.optim as optim

print("PyTorch: {}".format(torch.__version__))

def init_weights(m):
    if type(m) == nn.Linear:
        m.weight.data.uniform_()
        m.bias.data.uniform_()

def do_neural_stuff():
    # Initialize the network
    fcffnn = nn.Sequential(
            nn.Linear(1, 10),
            nn.Sigmoid(),
            nn.Linear(10, 10),
            nn.Sigmoid(),
            nn.Linear(10, 2)
            )
    fcffnn.apply(init_weights)

    # Run the network
    loss_f = nn.CrossEntropyLoss()
    opt = optim.Adam(fcffnn.parameters(), lr = 3e-4, weight_decay = 0.001)

    # Compute loss, backpropagate, adjust params along neg error gradient
    output = fcffnn(torch.tensor([0], dtype=torch.float))
    loss = loss_f(output.view(1, 2), torch.tensor([1], dtype=torch.long))
    fcffnn.zero_grad()
    loss.backward()
    opt.step()
    
# Run the network on the parent process
do_neural_stuff() # PyTorch works as expected when this line is commented out

def parallelModelRunnerProc(proc_nb, response_queue):
    print("Forked Process {}".format(proc_nb))
    do_neural_stuff()
    
    # This is important -- without the sleep(), the threads exit
    # without ever crashing due to a native error in this simplified example
    time.sleep(5)
    r_queue.put(True)

# 1) Run the network on 5 child processes
# 2) Wait for background activity (CUDA initialization -- even though it is not asked for)
# 3) Observe and report error
r_queue = Queue()
servers = []
nb_responses = 0
child_ct = 5
for proc_nb in range(child_ct):
    server = Process(target=parallelModelRunnerProc, args=(proc_nb, r_queue))
    server.start()
    servers.append(server)

while nb_responses < child_ct:
    nb_dead = sum([not x.is_alive() for x in servers])
    if not r_queue.empty():
        r_queue.get()
        nb_responses += 1
    
    if nb_dead > nb_responses:
        raise Exception("Something died unexpectedly")

```

**Program Output**
```
PyTorch: 1.4.0
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error (setDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:42)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe417f65193 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xd0c8 (0x7fe4181980c8 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: torch::autograd::Engine::set_device(int) + 0x159 (0x7fe3bd8a8fd9 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::Engine::thread_init(int) + 0x1a (0x7fe3bd8a995a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7fe40449891a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xedef (0x7fe418b80def in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x76db (0x7fe41ca896db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fe41c7b288f in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error (setDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:42)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe417f65193 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xd0c8 (0x7fe4181980c8 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: torch::autograd::Engine::set_device(int) + 0x159 (0x7fe3bd8a8fd9 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::Engine::thread_init(int) + 0x1a (0x7fe3bd8a995a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7fe40449891a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xedef (0x7fe418b80def in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x76db (0x7fe41ca896db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fe41c7b288f in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error (setDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:42)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe417f65193 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xd0c8 (0x7fe4181980c8 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: torch::autograd::Engine::set_device(int) + 0x159 (0x7fe3bd8a8fd9 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::Engine::thread_init(int) + 0x1a (0x7fe3bd8a995a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7fe40449891a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xedef (0x7fe418b80def in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x76db (0x7fe41ca896db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fe41c7b288f in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error (setDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:42)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe417f65193 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xd0c8 (0x7fe4181980c8 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: torch::autograd::Engine::set_device(int) + 0x159 (0x7fe3bd8a8fd9 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::Engine::thread_init(int) + 0x1a (0x7fe3bd8a995a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7fe40449891a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xedef (0x7fe418b80def in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x76db (0x7fe41ca896db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fe41c7b288f in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error (setDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:42)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe417f65193 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xd0c8 (0x7fe4181980c8 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: torch::autograd::Engine::set_device(int) + 0x159 (0x7fe3bd8a8fd9 in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::autograd::Engine::thread_init(int) + 0x1a (0x7fe3bd8a995a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7fe40449891a in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xedef (0x7fe418b80def in /home/nick.roth/ml_python/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x76db (0x7fe41ca896db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fe41c7b288f in /lib/x86_64-linux-gnu/libc.so.6)

Traceback (most recent call last):
  File "pytorch_crash.py", line 69, in <module>
    raise Exception("Something died unexpectedly")
Exception: Something died unexpectedly
```

## Expected behavior

PyTorch should be usable on both child and parent processes if the CUDA device is never used.

## Environment

```
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 435.21
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] Could not collect
```

CUDA version is 10.1 based on output from `nvidia-smi`.

PyTorch was installed on a virtualenv using pip. The machine used to reproduce this bug is an AWS p3.2xlarge instance with CUDA support provided by nvidia-driver-435 and nvidia-utils-435 packages.

The box was deployed in the `us-west-2` AWS region with the `ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190212.1 (ami-005bdb005fb00e791)` base image.

## Additional context

Similar reports to this exist, but they either lack reproducibility or are classified as user error since the programmer attempts to use CUDA on multiple processes in a tree. See #30900, #2517 and #3491. In our environment, using `spawn` instead of `fork` to create the child processes does not resolve the issue.



cc @ezyang @SsnL @albanD @zou3519 @gqchen @ngimel