Repository: pytorch
Issue: Using torch from multiple processes is slow
Labels: module: multiprocessing,triaged

### 🐛 Describe the bug

What is the correct way to use torch from multiple processes. I would expect the code in the spawned process below would run at the same speed throughout, but it is much slower whilst the main process is also doing torch things. This is a toy example, but my real code has a similar problem.

```python
# type: ignore
from contextlib import contextmanager
from time import perf_counter
from typing import Iterator

import torch
import torch.multiprocessing as mp

mp.set_start_method("spawn", force=True)


@contextmanager
def timer(name: str) -> Iterator[None]:
    start = perf_counter()
    yield
    end = perf_counter()
    print(f"{name}: {end - start:.5f}s")


def do_stuff():
    a = torch.empty((1_00_000, 2_000))
    b = torch.empty((1_00_000, 2_000))
    a.copy_(b[torch.randperm(b.size(0))])


def proc():
    for _ in range(20):
        with timer("p"):
            # NOTE this is slow while we are do_stuff'ing in the main process
            do_stuff()


if __name__ == "__main__":
    torch.set_num_threads(1)
    p = mp.Process(target=proc)
    p.start()
    torch.set_num_threads(5)
    for _ in range(30):
        with timer("m"):
            do_stuff()
    p.join()
```

My machine has 30 cores so I'd expect everything to work independantly of each other (as I'm only using 6 threads), but what I see is when `do_stuff` is called from `proc` while the main process is still in the `m` loop, then it takes about 1.6s, but when the `m` loop finishes it speeds up to 0.8s. 

```
m: 0.48198s
p: 1.62041s
m: 0.52251s
m: 0.54056s
m: 0.61196s
p: 1.83045s
m: 0.55137s
m: 0.53094s
m: 0.57954s
m: 0.50424s
p: 1.73075s
m: 0.56716s
m: 0.51699s
m: 0.50080s
p: 1.68981s
m: 0.53625s
m: 0.49879s
m: 0.49690s
p: 1.68669s
m: 0.55190s
m: 0.54055s
m: 0.52681s
p: 1.52235s
################ m loop finished
p: 0.83526s
p: 0.81527s
p: 0.81331s
p: 0.82571s
p: 0.81018s
p: 0.77474s
p: 0.75070s
p: 0.83396s
p: 0.89753s
p: 0.78945s
p: 0.79359s
p: 0.75883s
```

### Versions

```
>>> print(torch.__config__.show())
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
  ```

cc @VitalyFedyunin