Repository: pytorch
Issue: Tensor on shared memory is set to 0 when using concurrent.futures and CUDA
Labels: oncall: distributed,triaged

### 🐛 Describe the bug

The following code works perfectly on CPU. On CUDA, the second print shows that the weights are all 0. 

If I pass `l` to `goo` through the pool, then in the thread its weights are 0. Without the pool it works. 
This happens only on CUDA. 

```
import torch
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor
from torch import multiprocessing as mp

def goo(l):
    return l(torch.rand(2, device='cuda'))

def run():
    ctx = mp.get_context('spawn')

    pool1 = ProcessPoolExecutor(1)

    l = torch.nn.Linear(2, 2).to('cuda').share_memory()
    print(vars(l))

    pool1.submit(goo, l)

    def foo():
        print(vars(l))

    thread = ThreadPoolExecutor(1)
    thread.submit(foo)

if __name__ == '__main__':
    run()

```

### Versions

This is on a Windows computer, but the issue happens also on a SLURM-based cluster with older version of python and pytorch. 

Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.9 | packaged by conda-forge | (main, Jan 11 2023, 15:15:40) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 12.0.76
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650 with Max-Q Design
Nvidia driver version: 527.41
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2000
DeviceID=CPU0
Family=198
L2CacheSize=7680
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2000
Name=12th Gen Intel(R) Core(TM) i7-1280P
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] numpy-quaternion==2022.4.2
[pip3] pytorch-lightning==2.0.2
[pip3] pytorch-memlab==0.2.4
[pip3] torch==2.0.0
[pip3] torch-tb-profiler==0.4.1
[pip3] torchaudio==2.0.0
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.15.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           haa95532_640
[conda] mkl-service               2.4.0           py310h2bbff1b_0
[conda] mkl_fft                   1.3.1           py310ha0764ea_0
[conda] mkl_random                1.2.2           py310h4ed8f06_0
[conda] numpy                     1.23.5          py310h60c9a35_0
[conda] numpy-base                1.23.5          py310h04254f7_0
[conda] numpy-quaternion          2022.4.2                 pypi_0    pypi
[conda] pytorch                   2.0.0           py3.10_cuda11.7_cudnn8_0    pytorch
[conda] pytorch-cuda              11.7                 h16d0643_3    pytorch
[conda] pytorch-lightning         2.0.2                    pypi_0    pypi
[conda] pytorch-memlab            0.2.4                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.13.1                   pypi_0    pypi
[conda] torch-tb-profiler         0.4.1                    pypi_0    pypi
[conda] torchaudio                2.0.0                    pypi_0    pypi
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchvision               0.15.0                   pypi_0    pypi

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu