Repository: pytorch
Issue: Unexpected behaviour with shared modules in multiprocessing on WSL2
Labels: module: multiprocessing,triaged,module: wsl

### 🐛 Describe the bug

When sharing a cuda module between processes in WSL2, the module weights seem to be initialized to zero for newly created processes.

```python
# debug.py script
import torch


def test(model):
    print(model.weight)  # Prints 0, should print -0.0075
    model.load_state_dict({"weight": torch.tensor([[1.0]]), "bias": torch.tensor([0.0])})
    print(model.weight)  # Prints 1


if __name__ == "__main__":
    torch.manual_seed(0)
    device = torch.device("cuda")
    model = torch.nn.Linear(1, 1).to(device)
    model.share_memory()
    print(model.weight)  # Prints -0.0075
    ctx = torch.multiprocessing.get_context("spawn")
    proc = ctx.Process(target=test, args=(model,))
    proc.start()  # See test(model)
    proc.join()
    print(model.weight)  # Prints 1, model is indeed shared across processes
```

Expected behaviour:
Linear layer weight is -0.0075 both in the main process and in the new process. After updating the weights, it is 1 in both processes.

Actual behaviour:
Linear layer weight is 0 in the new process immediately after sharing. Updating changes the weights in both processes, so the module still seems to be shared properly.

This is only an issue for cuda modules. If you change the device to "cpu", the above code works as expected. Surprisingly, the behaviour causes even more trouble in Docker. I have created a minimal example using docker compose.

docker-compose.yaml:
```
version: "3.8"
services:
  debug_container:
    build:
      dockerfile: Dockerfile
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]
```

Dockerfile
```
FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime

COPY debug.py /home/debug.py

ENTRYPOINT ["python", "/home/debug.py"]
```

Create a new folder and copy the above content into the docker compose file, the dockerfile and the debug.py script. Running `docker compose up` fails with a CUDA RuntimeError

```
 ✔ Container debug-debug_container-1  Recreated                                                                                                                                                                                                                                        0.0s 
Attaching to debug-debug_container-1
debug-debug_container-1  | Parameter containing:
debug-debug_container-1  | tensor([[-0.0075]], device='cuda:0', requires_grad=True)
debug-debug_container-1  | Traceback (most recent call last):
debug-debug_container-1  |   File "<string>", line 1, in <module>
debug-debug_container-1  |   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
debug-debug_container-1  |     exitcode = _main(fd, parent_sentinel)
debug-debug_container-1  |   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
debug-debug_container-1  |     self = reduction.pickle.load(from_parent)
debug-debug_container-1  |   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 147, in rebuild_cuda_tensor
debug-debug_container-1  |     storage = storage_cls._new_shared_cuda(
debug-debug_container-1  |   File "/opt/conda/lib/python3.10/site-packages/torch/storage.py", line 1085, in _new_shared_cuda
debug-debug_container-1  |     return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)
debug-debug_container-1  | RuntimeError: CUDA error: invalid resource handle
debug-debug_container-1  | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
debug-debug_container-1  | For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
debug-debug_container-1  | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
debug-debug_container-1  | 
debug-debug_container-1  | Parameter containing:
debug-debug_container-1  | tensor([[0.]], device='cuda:0', requires_grad=True)
debug-debug_container-1  | [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
debug-debug_container-1 exited with code 0
```

I am using a similar pattern in a Docker project that was working until recently and now started crashing, so whatever causes the issue must have been introduced in the last few weeks.

### Versions


PyTorch version: 2.1.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:40:35) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 536.99
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen 9 7950X 16-Core Processor
CPU family:                      25
Model:                           97
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        2
BogoMIPS:                        8982.94
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm
Virtualization:                  AMD-V
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        16 MiB (16 instances)
L3 cache:                        32 MiB (1 instance)
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.1
[pip3] torch==2.1.0
[pip3] triton==2.1.0
[conda] Could not collect

cc @VitalyFedyunin