Repository: pytorch
Issue: Parameters of cuda module zero out when used in multiprocessing
Labels: module: docs,module: multiprocessing,module: cuda,triaged

### 🐛 Describe the bug

Inspecting the parameters of a module after it has been used in a process or pool reveals that the parameters become zero. I cannot find documentation saying this is intended behavior or a known antipattern.

Example:
```
import torch.nn as nn
import torch.multiprocessing as mp
def do_nothing(module):
    return

if __name__ == "__main__":
    mp.set_start_method('spawn')
    module = nn.Linear(4, 4)
    module.cuda()
    module.share_memory() # as far as I can tell this is a no-op?

    print(next(module.parameters()))
    p = mp.Process(target=do_nothing, args=(module,))
    p.start()
    p.join()
    print(next(module.parameters()))
```

Output:
```
Parameter containing:
tensor([[ 0.2924,  0.0857, -0.1374,  0.0929],
        [-0.4918, -0.2784, -0.3998, -0.0934],
        [ 0.1911, -0.3074,  0.0238, -0.1824],
        [-0.0655,  0.1752,  0.0798, -0.4360]], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]], device='cuda:0', requires_grad=True)
```
A workaround is to keep the module on cpu, then transfer it onto cuda inside each process.

First found on torch 1.12.1. Persists in torch 2.0.1.
Thanks!

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:29:51) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060
Nvidia driver version: 536.23
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3501
DeviceID=CPU0
Family=107
L2CacheSize=3072
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=3501
Name=AMD Ryzen 5 5600 6-Core Processor
ProcessorType=3
Revision=8450

Versions of relevant libraries:
[pip3] numpy==1.23.2
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0              16_win64_mkl    conda-forge
[conda] cudatoolkit               11.6.0              hc0ea762_10    conda-forge
[conda] libblas                   3.9.0              16_win64_mkl    conda-forge
[conda] libcblas                  3.9.0              16_win64_mkl    conda-forge
[conda] liblapack                 3.9.0              16_win64_mkl    conda-forge
[conda] liblapacke                3.9.0              16_win64_mkl    conda-forge
[conda] mkl                       2022.1.0           h6a75c08_874    conda-forge
[conda] mkl-devel                 2022.1.0           h57928b3_875    conda-forge
[conda] mkl-include               2022.1.0           h6a75c08_874    conda-forge
[conda] numpy                     1.23.2          py310h8a5b91a_0    conda-forge
[conda] pytorch                   2.0.1           py3.10_cuda11.7_cudnn8_0    pytorch
[conda] pytorch-cuda              11.7                 h16d0643_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi

cc @svekars @carljparker @VitalyFedyunin @ptrblck