Repository: pytorch
Issue: `torch.distributed.pipelining` hang and timeout in CPU gloo backend
Labels: oncall: distributed,triaged,module: pipelining

### 🐛 Describe the bug

When executing the [pippy_bert.py](https://github.com/pytorch/PiPPy/blob/main/examples/huggingface/pippy_bert.py) example with cpu gloo backend:
```
torchrun --nproc-per-node 4 pippy_bert.py
```
I got the following hang and timeout error:
```
Pipeline stage 3 21M params
Pipeline stage 1 21M params
Pipeline stage 0 45M params
Pipeline stage 2 21M params
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 108, in <module>
[rank0]:     run(args)
[rank0]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 72, in run
[rank0]:     schedule.step(**inputs)
[rank0]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 599, in step
[rank0]:     self._step_microbatches(args_split, kwargs_split, targets_split, losses)
[rank0]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 708, in _step_microbatches
[rank0]:     work.wait()
[rank0]: RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:525] Read error [127.0.1.1]:42417: Connection reset by peer
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 108, in <module>
[rank2]:     run(args)
[rank2]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 74, in run
[rank2]:     out = schedule.step()
[rank2]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 599, in step
[rank2]:     self._step_microbatches(args_split, kwargs_split, targets_split, losses)
[rank2]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 690, in _step_microbatches
[rank2]:     work.wait()
[rank2]: RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [127.0.1.1]:42417
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 108, in <module>
[rank1]:     run(args)
[rank1]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 74, in run
[rank1]:     out = schedule.step()
[rank1]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 599, in step
[rank1]:     self._step_microbatches(args_split, kwargs_split, targets_split, losses)
[rank1]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 690, in _step_microbatches
[rank1]:     work.wait()
[rank1]: RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 108, in <module>
[rank3]:     run(args)
[rank3]:   File "/home/hongrongh/PiPPy/examples/huggingface/pippy_bert.py", line 74, in run
[rank3]:     out = schedule.step()
[rank3]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 599, in step
[rank3]:     self._step_microbatches(args_split, kwargs_split, targets_split, losses)
[rank3]:   File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 690, in _step_microbatches
[rank3]:     work.wait()
[rank3]: RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [127.0.0.1]:1710
W0805 17:17:32.369000 2465323 torch/distributed/elastic/multiprocessing/api.py:891] Sending process 2465359 closing signal SIGTERM
E0805 17:17:32.433000 2465323 torch/distributed/elastic/multiprocessing/api.py:863] failed (exitcode: 1) local_rank: 0 (pid: 2465356) of binary: /home/hongrongh/PiPPy/myvenv/bin/python3
Traceback (most recent call last):
  File "/home/hongrongh/PiPPy/myvenv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/hongrongh/PiPPy/myvenv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
pippy_bert.py FAILED
------------------------------------------------------------
```
BTW, [pippy_gpt2.py](https://github.com/pytorch/PiPPy/blob/main/examples/huggingface/pippy_gpt2.py) works but [pippy_llama.py](https://github.com/pytorch/PiPPy/blob/main/examples/llama/pippy_llama.py) is also timeout.

### Versions

Collecting environment information...
PyTorch version: 2.5.0.dev20240805+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      43 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             32
On-line CPU(s) list:                0-31
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen 9 3950X 16-Core Processor
CPU family:                         23
Model:                              113
Thread(s) per core:                 2
Core(s) per socket:                 16
Socket(s):                          1
Stepping:                           0
Frequency boost:                    enabled
CPU max MHz:                        3500.0000
CPU min MHz:                        2200.0000
BogoMIPS:                           6986.85
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization:                     AMD-V
L1d cache:                          512 KiB (16 instances)
L1i cache:                          512 KiB (16 instances)
L2 cache:                           8 MiB (16 instances)
L3 cache:                           64 MiB (4 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pytorch-triton==3.0.0+45fff310c8
[pip3] torch==2.5.0.dev20240805+cpu
[pip3] triton==2.3.1
[conda] Could not collect

My `transformers` version is `4.37.1`

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o