Repository: pytorch
Issue: distributed/test_jit_c10d.py fails with RuntimeError: Address already in use
Labels: oncall: distributed

## 🐛 Bug

distributed/test_jit_c10d.py fails with RuntimeError: Address already in use

## To Reproduce

Steps to reproduce the behavior:

1. Install EasyBuild on a CentOS 7.9 (el7) system.
2. Build PyTorch: $ eb --from-pr=12814 -r  PyTorch-1.8.1-fosscuda-2020b.eb 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

The EasyBuild log file ends with:

ERROR: test_process_group_nccl_serialization (__main__.ProcessGroupNCCLJitTest)

Traceback (most recent call last):
  File "/tmp/eb-nk9KGi/tmpUJ_a1q/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 146, in wrapper
    return func(*args, **kwargs)
  File "distributed/test_jit_c10d.py", line 143, in test_process_group_nccl_serialization
    pg_nccl = self._create_nccl_pg("nccl_process_group_as_module_member")
  File "distributed/test_jit_c10d.py", line 60, in _create_nccl_pg
    tcp_store = _create_tcp_store()
  File "distributed/test_jit_c10d.py", line 40, in _create_tcp_store
    return torch.classes.dist_c10d.TCPStore(addr, port, 1, True, timeout_millisecond)
RuntimeError: Address already in use

----------------------------------------------------------------------
Ran 9 tests in 0.455s

FAILED (errors=1)
Traceback (most recent call last):
  File "run_test.py", line 926, in <module>
    main()
  File "run_test.py", line 905, in main
    raise RuntimeError(err_message)
RuntimeError: distributed/test_jit_c10d failed!
 (at easybuild/tools/run.py:565 in parse_cmd_output)
== 2021-06-03 15:01:04,392 filetools.py:1876 INFO Removing lock /home/modules/software/.locks/_home_modules_software_PyTorch_1.8.1-fosscuda-2020b.lock...
== 2021-06-03 15:01:04,393 filetools.py:358 INFO Path /home/modules/software/.locks/_home_modules_software_PyTorch_1.8.1-fosscuda-2020b.lock successfully removed.
== 2021-06-03 15:01:04,394 filetools.py:1880 INFO Lock removed: /home/modules/software/.locks/_home_modules_software_PyTorch_1.8.1-fosscuda-2020b.lock
== 2021-06-03 15:01:04,394 easyblock.py:3643 WARNING build failed (first 300 chars): cmd "export PYTHONPATH=/tmp/eb-nk9KGi/tmpUJ_a1q/lib/python3.8/site-packages:$PYTHONPATH &&  cd test && PYTHONUNBUFFERED=1 /home/modules/software/Python/3.8.6-GCCcore-10.2.0/bin/python run_test.py --verbose -x distributed/rpc/test_process_group_agent test_quantization " exited with exit code 1 and ou
== 2021-06-03 15:01:04,395 easyblock.py:300 INFO Closing log for application name PyTorch version 1.8.1

## Expected behavior

A PyTorch software module should be built and tested successfully.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)

$ python collect_env.py
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.2.5

Python version: 2.7 (64-bit runtime)
Python platform: Linux-3.10.0-1160.25.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090
GPU 8: NVIDIA GeForce RTX 3090
GPU 9: NVIDIA GeForce RTX 3090

Nvidia driver version: 465.24.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect

## Additional context

There are no other network services running on this system, which could potentially grab the TCP port.  The error has been reproducible at least 3 times.

I notice that on el7 (CentOS 7.9 here) generally, a service releasing a TCP port will not make that port available until after a couple of minutes.  Maybe that behavior is messing up this failing PyTorch test?


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23 @gmagogsfm