Repository: pytorch
Issue: Memory leak when using share_memory_ on cuda device
Labels: module: memory usage,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

I'd like multiple processes use the same tensor in a read-only operation using multiprocessing. When I set `share_memory_()` for a tensor and pass that to a child process, the memory scales linearly with the number of child process if cuda device is used. If I only use cpu for this example, the memory consumption is constant.


## To Reproduce

Steps to reproduce the behavior:

In the following code sample, if you comment out `c.to(device)`, then the behavior is as expected where in the memory doesn't scale with the number of child processes.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
```
import numpy as np
from multiprocessing import shared_memory, get_context
import time
import torch
import copy

dim = 100
batch_size = 10
sleep_time = 2
npe = 4  # number of parallel executions

# cuda
if torch.cuda.is_available():
    dev = 'cuda:0'
else:
    dev = "cpu"
device = torch.device(dev)


def step(i, arr):
    b = arr[i * batch_size: (i + 1) * batch_size, :]
    c = copy.deepcopy(b)
    
    # comment the following line and memory consumption DOES NOT scale with
    # number of child processes and is constant
    c = c.to(device)

    time.sleep(sleep_time)


if __name__ == '__main__':
    arr = torch.randn((dim, dim))
    arr.share_memory_()

    # create list of inputs to be executed in parallel
    inp = [(x, arr) for x in range(npe)]

    # sleep added before and after launching multiprocessing to monitor the memory consumption
    print('before pool')  # to check memory with top or htop
    time.sleep(sleep_time)

    context = get_context('spawn')
    with context.Pool(npe) as pool:
        print('after pool')  # to check memory with top or htop
        time.sleep(sleep_time)

        pool.starmap(step, inp)

    time.sleep(sleep_time)

```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

I expect constant memory (RAM) consumption because the tensor `arr` is shared. If `.to(device)` is not used, expected behavior is seen. But when I try to send any tensor to cuda device, it seems like data is copied for every process.

Is this expected?

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
PyTorch version: 1.7.0a0+3f5eee6
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 440.100
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] numpy==1.19.1
[pip] torch==1.7.0a0
[pip] torchvision==0.8.0a0+15848ed
[conda] blas                      1.0                         mkl  
[conda] magma-cuda100             2.5.2                         1    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.1           py38hbc911f0_0  
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] torch                     1.7.0a0                  pypi_0    pypi
[conda] torchvision               0.8.0a0+15848ed          pypi_0    pypi

## Additional context

<!-- Add any other context about the problem here. -->