Repository: pytorch
Issue: memory leaking when doing all_to_all_single communication 
Labels: oncall: distributed,triaged,module: c10d

### 🐛 Describe the bug

I observed memory-leaking problem (CPU memory is always decreasing) when training an moe model on (pytorch 1.10, CUDA 11.3, NCCL 2 .10.3). I found that the problem comes from `all_to_all_single` API. But the problem could not be found on (pytorch 1.9.1, CUDA 11.1, NCCL 2 .7.08)

The following script could reproduce the problem. 

```python
import torch
import os
import torch.multiprocessing as mp
import torch.distributed as dist

def setup(rank, world_size):
    os.environ['MASTER_ADDR']='127.0.0.2'
    os.environ['MASTER_PORT']="123457"
    dist.init_process_group('nccl', rank=rank, world_size=world_size)

def demo(rank, world_size):
    setup(rank, world_size)
    x=torch.zeros(int(1e6)).to(rank)
    output=torch.empty_like(x)
    
    while True:
        if dist.get_rank()==0:
            print('running all2all communication')
        dist.all_to_all_single(output, x)
    
if __name__=='__main__':
    world_size=8
    mp.spawn(
        demo,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )
```

### Versions (With Memory leaking)

Collecting environment information...
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.6 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.2
Libc version: glibc-2.10

Python version: 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-99-generic-x86_64-with-debian-stretch-sid
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
GPU 4: NVIDIA RTX A6000
GPU 5: NVIDIA RTX A6000
GPU 6: NVIDIA RTX A6000
GPU 7: NVIDIA RTX A6000

Nvidia driver version: 470.86
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    defaults
[conda] faiss-gpu                 1.7.2           py3.7_h28a55e0_0_cuda11.3    pytorch
[conda] libfaiss                  1.7.2           hfc2d529_0_cuda11.3    pytorch
[conda] mkl                       2021.4.0           h8d4b97c_729    conda-forge
[conda] mkl-service               2.4.0            py37h402132d_0    conda-forge
[conda] mkl_fft                   1.3.1            py37h3e078e5_1    conda-forge
[conda] mkl_random                1.2.2            py37h219a48f_0    conda-forge
[conda] numpy                     1.21.2           py37h20f2e39_0    defaults
[conda] numpy-base                1.21.2           py37h79a1101_0    defaults
[conda] pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch

### Versions (Without memory leaking)
Collecting environment information...
PyTorch version: 1.9.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.17

Python version: 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-171-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
GPU 4: NVIDIA RTX A6000
GPU 5: NVIDIA RTX A6000
GPU 6: NVIDIA RTX A6000
GPU 7: NVIDIA RTX A6000

Nvidia driver version: 470.129.06
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.1+cu111
[conda] numpy                     1.19.5                    <pip>
[conda] torch                     1.9.1+cu111               <pip>

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501