Repository: pytorch
Issue: High GPU context memory on Torch 1.11.0 but none on Torch 1.10.0
Labels: needs reproduction,module: cuda,triaged

### 🐛 Describe the bug

I am running a multi-gpu code on 2 GPUs and tested it with Torch 1.10.0 and Torch 1.11.0. While using Torch 1.10.0 I only see 2 processes (1 on each GPU), while running it on Torch 1.11.0 I can see 3 processes (2 on GPU 0, and 1 on GPU 1). The additional process on GPU 0 has same pid as GPU 1 and takes around 500 MB of space. I believe it is because of the context memory as per my search on the forums. But wanted to understand why it only occurs with Torch 1.11.0 and not with Torch 1.10.0. The details are mentioned below.

Torch 1.10.0: 
Docker Image: nvcr.io/nvidia/pytorch:21.09-py3 (CUDA 11.4)
Memory usage:
|    0   N/A  N/A   1123607      C   /opt/conda/bin/python            4521MiB |
|    1   N/A  N/A   1123611      C   /opt/conda/bin/python            4501MiB |

Torch 1.11.0: 
Docker Image: nvcr.io/nvidia/pytorch:21.11-py3 (CUDA 11.5)
Memory usage:
|    0   N/A  N/A   1418076      C   /opt/conda/bin/python            4537MiB |
|    0   N/A  N/A   1418077      C   /opt/conda/bin/python             543MiB |
|    1   N/A  N/A   1418077      C   /opt/conda/bin/python            4537MiB |


### Versions

Collecting environment information...
PyTorch version: 1.11.0a0+b6df043
Is debug build: False
CUDA used to build PyTorch: 11.5
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.21.3
Libc version: glibc-2.31

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.5.50
GPU models and configuration:
GPU 0: NVIDIA GeForce GTX 1070
GPU 1: NVIDIA GeForce GTX 1070

Nvidia driver version: 510.73.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.4
[pip3] nvidia-dlprof-pytorch-nvtx==1.7.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.11.0a0+b6df043
[pip3] torch-tensorrt==1.0.0a0
[pip3] torchtext==0.12.0a0
[pip3] torchvision==0.11.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-include               2019.5                      281    conda-forge
[conda] numpy                     1.21.4           py38he2449b9_0    conda-forge
[conda] nvidia-dlprof-pytorch-nvtx 1.7.0                    pypi_0    pypi
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.11.0a0+b6df043          pypi_0    pypi
[conda] torch-tensorrt            1.0.0a0                  pypi_0    pypi
[conda] torchtext                 0.12.0a0                 pypi_0    pypi
[conda] torchvision               0.11.0a0                 pypi_0    pypi


cc @ngimel