Repository: pytorch
Issue: [Bug] [CUDA IPC] CUDA IPC memory cost
Labels: module: performance,module: multiprocessing,module: cuda,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
In CUDA IPC, CPU memory consumption is too high

## To Reproduce

Steps to reproduce the behavior:

1. Run the code below
2. Profile memory usage of the child process

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
```
import torch
import torch.multiprocessing as mp
import time


def child(queue):
    tensor = queue.get()
    time.sleep(50)


if __name__ == '__main__':
    mp.set_start_method('spawn')
    a = torch.zeros(1000000, 100)
    a = a.to(0)
    queue = mp.Queue(4)
    proc = mp.Process(target=child, args=(queue,))
    proc.start()
    queue.put(a)
    proc.join()
```

![image](https://user-images.githubusercontent.com/41138939/133512796-2746de71-8fd2-40a6-9fa7-e4b9fd25a7cc.png)

When running my application code to spawn child processes with large CUDA tensors, CPU memory increased quickly and higher than the shared tensor size. 

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
CPU memory should be no more than the tensor size.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.2 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.21.1
Libc version: glibc-2.26

Python version: 3.7.8 (default, Nov 26 2020, 03:13:20)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-142-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: TITAN X (Pascal)
GPU 1: TITAN X (Pascal)
GPU 2: TITAN X (Pascal)
GPU 3: TITAN X (Pascal)

Nvidia driver version: 430.26
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.7.0
[pip3] torch-cluster==1.5.8
[pip3] torch-geometric==1.6.1
[pip3] torch-scatter==2.0.5
[pip3] torch-sparse==0.6.11
[pip3] torch-spline-conv==1.2.0
[pip3] torchvision==0.8.1

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin @ngimel @heitorschueroff