Repository: pytorch
Issue: destroy_process_group() does not clear worker count, leading to "Timed out initializing process group in store based barrier"
Labels: oncall: distributed

## 🐛 Bug

I am fine-tuning a series of huggingface transformer models with their `Trainer` class, using DDP and the pytorch `torchrun` script to distribute the training across 4 GPUs. It trains the first model fine, but the script hangs and times out when trying to initialise the torch process group on the second iteration.

Upon further investigation, this appears to be a pytorch bug, specifically with the TCPStore worker count not being cleared when calling `destroy_process_group()`.

## To Reproduce

Steps to reproduce the behavior:

Here is some distilled code that reproduces the behaviour:

```python3
import torch.distributed as dist
import os
from datetime import timedelta


def setup():
    dist.init_process_group(
        "gloo",
        timeout=timedelta(seconds=10)  # can be removed, but it makes it quicker to see the timeout
    )


def teardown():
    dist.destroy_process_group()


def run():
    setup()

    # Run experiment
    print(f"Running rank {os.getenv('RANK')}")

    teardown()


def main():
    for _ in range(5):
        run()


if __name__ == "__main__":
    raise SystemExit(main())

```

To launch this code I am using:
```sh
$ python -m torch.distributed.run --nnodes=1 --nproc_per_node=1 test_lockup.py
```

Here is the output I am getting:

```
[W ProcessGroupGloo.cpp:684] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
Running rank 0
Traceback (most recent call last):
  File "experiments/test_lockup.py", line 38, in <module>
    raise SystemExit(main())
  File "experiments/test_lockup.py", line 34, in main
    run()
  File "experiments/test_lockup.py", line 24, in run
    setup()
  File "experiments/test_lockup.py", line 7, in setup
    dist.init_process_group(
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 561, in init_process_group
    _store_based_barrier(rank, store, timeout)
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 221, in _store_based_barrier
    raise RuntimeError(
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=1, worker_count=2, timeout=0:00:10)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 94325) of binary: /Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/bin/python
/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:367: UserWarning: 

**********************************************************************
               CHILD PROCESS FAILED WITH NO ERROR_FILE                
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 94325 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:

  from torch.distributed.elastic.multiprocessing.errors import record

  @record
  def trainer_main(args):
     # do train
**********************************************************************
  warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
  File "/Users/lnethert/.pyenv/versions/3.8.9/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/lnethert/.pyenv/versions/3.8.9/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/run.py", line 702, in <module>
    main()
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 361, in wrapper
    return f(*args, **kwargs)
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/run.py", line 698, in main
    run(args)
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
    elastic_launch(
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/Users/lnethert/.local/share/virtualenvs/modelling-H-HfK1ma/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
***************************************
   experiments/test_lockup.py FAILED   
=======================================
Root Cause:
[0]:
  time: 2021-10-13_11:35:30
  rank: 0 (local_rank: 0)
  exitcode: 1 (pid: 94325)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
  <NO_OTHER_FAILURES>
***************************************
```

## Expected behavior

I expected the output to be this:

```
[W ProcessGroupGloo.cpp:684] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
Running rank 0
Running rank 0
Running rank 0
Running rank 0
Running rank 0
```


## Environment

```
Collecting environment information...
PyTorch version: 1.9.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 10.15.2 (x86_64)
GCC version: Could not collect
Clang version: 11.0.3 (clang-1103.0.32.62)
CMake version: Could not collect
Libc version: N/A

Python version: 3.8 (64-bit runtime)
Python platform: macOS-10.15.2-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.9.1
[pip3] torchvision==0.10.1
[conda] Could not collect
```

## Additional context

IMO, the most salient message in the output is this line:

```
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=1, worker_count=2, timeout=0:00:10)
```

Specifically, `world_size=1, worker_count=2`. Note that I am launching with `--nproc_per_node=1`, so the worker_count should never exceed 1.


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang