Repository: pytorch
Issue: Multiprocessing - shared memory
Labels: module: multiprocessing,triaged

### 🐛 Describe the bug

Hello,

Is the following to be expected for Tensors? Note, I am not using torch.multiprocessing and I am not calling 'share_memory_':

<b> outside.py: </b>
```python
def train(data):
    print(data.is_shared()) # Prints True ?!
    data[:3] = 999
```

<b> Training.ipynb </b>:
```python
import multiprocessing as mp # Note: NOT importing torch.multiprocessing
from outside import train # External export needed in Notebooks due to multiprocessing error
import torch

data = torch.tensor([1,3,5,7])
## Commenting out call to 'share_memory_'
#     data.share_memory_()
print(data.is_shared()) # Prints False as expected
processes = []
num_processes = 2
for rank in range(num_processes):
    p = mp.Process(target=train, args=(data,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()
print(data) 
# tensor([999, 999, 999, 7]) ?! Tensor is modified by train
```


Also, while we're at it, could someone be so kind and explain to me how the sharing of memory actually happens in PyTorch? From my understanding, torch.multiprocessing should provide a 'true' memory sharing capability, which should not copy objects over to child processes as opposed to how a Manager().dict() would work. In theory, this behavior happens after the __init__.py of the package is called, as special reducers for mp.Queue are registered on the ForkingPickler. However, I was not able to confirm whether this is actually the case from my set of experiments (id'ing everything from the tensor to the underlying storage objects results in different outputs, the same goes for calling 'data_ptr' on the tensor/storage; data within tensors remains consistent even when torch.multiprocessing is not explicitly imported). This is why I stumbled upon the 'is_shared' method, but unfortunately this seems to offer unexpected outputs (at least in my opinion), as shown above.

The reason why I am asking all these questions is because I would like to ensure my model is always shared among subprocesses because the memory demand of my project is quite big. A link to a similar question I posted on the Discussions website can be found here: https://discuss.pytorch.org/t/multiprocessing-shared-memory/139534. Thank you for your time!


### Versions

```
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 15:50:08) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 850M
Nvidia driver version: 496.49
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] neptune-pytorch-lightning==0.9.7
[pip3] numpy==1.21.1
[pip3] pytorch-lightning==1.4.0
[pip3] torch==1.9.0
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==1.7.2
[pip3] torch-scatter==2.0.8
[pip3] torch-sparse==0.6.11
[pip3] torch-spline-conv==1.2.1
[pip3] torchaudio==0.9.0
[pip3] torchdiffeq==0.2.2
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.10.0
[conda] blas                      2.112                       mkl    conda-forge
[conda] blas-devel                3.9.0              12_win64_mkl    conda-forge
[conda] cudatoolkit               11.1.1               heb2d755_7    conda-forge
[conda] libblas                   3.9.0              12_win64_mkl    conda-forge
[conda] libcblas                  3.9.0              12_win64_mkl    conda-forge
[conda] liblapack                 3.9.0              12_win64_mkl    conda-forge
[conda] liblapacke                3.9.0              12_win64_mkl    conda-forge
[conda] mkl                       2021.4.0           h0e2418a_729    conda-forge
[conda] mkl-devel                 2021.4.0           h57928b3_730    conda-forge
[conda] mkl-include               2021.4.0           h0e2418a_729    conda-forge
[conda] mypy_extensions           0.4.3            py38haa244fe_4    conda-forge
[conda] neptune-pytorch-lightning 0.9.7                    pypi_0    pypi
[conda] numpy                     1.21.1           py38h09042cb_0    conda-forge
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8_0    pytorch
[conda] pytorch-cluster           1.5.9           py38_torch_1.9.0_cu111    rusty1s
[conda] pytorch-geometric         1.7.2           py38_torch_1.9.0_cu111    rusty1s
[conda] pytorch-lightning         1.4.0              pyhd8ed1ab_0    conda-forge
[conda] pytorch-scatter           2.0.8           py38_torch_1.9.0_cu111    rusty1s
[conda] pytorch-sparse            0.6.11          py38_torch_1.9.0_cu111    rusty1s
[conda] pytorch-spline-conv       1.2.1           py38_torch_1.9.0_cu111    rusty1s
[conda] torchaudio                0.9.0                      py38    pytorch
[conda] torchdiffeq               0.2.2              pyhd8ed1ab_0    conda-forge
[conda] torchmetrics              0.6.0              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.10.0                   pypi_0    pypi
```

cc @VitalyFedyunin