Repository: pytorch
Issue: dist.destroy_process_group() don't release master_port when there're at least 2 process groups
Labels: oncall: distributed,module: c10d

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

Run the following code:
```python
import os, time
import torch.distributed as dist

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12345'

dist.init_process_group(backend='nccl', rank=0, world_size=1) # init default PG
ng = dist.new_group(backend='gloo') # create another PG
# ng = dist.new_gorup(backend='nccl') # this has the same result

if 1: # use if 0 will have the same result
    dist.destroy_process_group() # won't release master_port
else:
    dist.destroy_process_group(ng) # won't release master_port
    dist.destroy_process_group() # won't release master_port

# time.sleep(100) # won't change the result
dist.init_process_group(backend='nccl', rank=0, world_size=1) # will raise 'Address already in use' error
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

I expect the master_port be released, and re-init can succeed.

<!-- A clear and concise description of what you expected to happen. -->

## Environment
```
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A
OS: CentOS Linux release 8.1.1911 (Core)  (x86_64)
GCC version: (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Clang version: 9.0.1 (Red Hat 9.0.1-2.module_el8.2.0+309+0c7b6b03)
CMake version: version 3.11.4
Python version: 3.9.2 (releaselevel=final, serial=0, 64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 418.87.00
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.7.6.5
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.8.1
[pip3] torchaudio==0.8.0a0+e4e171a
[pip3] torchvision==0.2.2
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               10.1.243             h6bb024c_0    defaults
[conda] mkl                       2021.3.0           h726a3e6_557    conda-forge
[conda] mkl-service               2.4.0            py39h3811e60_0    conda-forge
[conda] mkl_fft                   1.3.0            py39h42c9631_2    defaults
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge
[conda] numpy                     1.20.1                   pypi_0    pypi
[conda] numpy-base                1.20.3           py39h74d4b33_0    defaults
[conda] pytorch                   1.8.1           py3.9_cuda10.1_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.8.1                      py39    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch
```
cc: @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23 @H-Huang 

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23