Invalid socket warning when running on SLURM
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When training on a single node, multi-gpu setup using SLURM, there's a warning that pops up that I cannot figure out:
```
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).
```

**Steps/Code to reproduce bug**

Using the [ASR fine-tuning example](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text_finetune.py) in a `sbatch` script:

```sh
[...]
srun python speech_to_text_finetune.py
[...]
```

It produces the following:
```
[...]
Using bfloat16 Automatic Mixed Precision (AMP)                                                                                                                                                                                                                                           
GPU available: True (cuda), used: True                                                                                                                                                                                                                                                   
TPU available: False, using: 0 TPU cores                                                                                                                                                                                                                                                 
IPU available: False, using: 0 IPUs                                                                                                                                                                                                                                                      
HPU available: False, using: 0 HPUs                                                                                                                                                                                                                                                      
Running in `fast_dev_run` mode: will run the requested loop using 20 batch(es). Logging and checkpointing is suppressed.                                                                                                                                                                 
Initializing distributed: GLOBAL_RANK: 5, MEMBER: 6/8                                                                                                                                                                                                                                    
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
Initializing distributed: GLOBAL_RANK: 4, MEMBER: 5/8                                                                                       
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                                                                                                                                                              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
Initializing distributed: GLOBAL_RANK: 7, MEMBER: 8/8                                                                                                                                                                                                                                    
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).              
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
[W socket.cpp:601] [c10d] The client socket has failed to connect to [ip-26-0-146-37]:22456 (errno: 22 - Invalid argument).                 
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/8                                                                                       
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/8                                                                                                                                                                                                                                    
Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/8                                                                                       
Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/8                                                                                                                                                                                                                                    
Initializing distributed: GLOBAL_RANK: 6, MEMBER: 7/8                                                                                       
----------------------------------------------------------------------------------------------------                                                                                                                                                                                     
distributed_backend=nccl                                                                                                                    
All distributed processes registered. Starting with 8 processes                                                                                                                                                                                                                          
----------------------------------------------------------------------------------------------------                                        
                                                                                                                                                                                                                                                                                         
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                                                                                                                                                                  
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                                                                                                                                                                  
LOCAL_RANK: 5 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
LOCAL_RANK: 4 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
LOCAL_RANK: 6 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
LOCAL_RANK: 7 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]                                                                                     
                                                                                                                                            
  | Name              | Type                              | Params                                                                          
------------------------------------------------------------------------                                                   
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0                                                                               
1 | encoder           | ConformerEncoder                  | 121 M                                                                           
2 | decoder           | ConvASRDecoder                    | 66.2 K                                                                          
3 | loss              | CTCLoss                           | 0                                                                               
4 | spec_augmentation | SpectrogramAugmentation           | 0                                                                               
5 | _wer              | WERBPE                            | 0                                                                                                                                                                                                                            
------------------------------------------------------------------------ 
[...]
```

**Expected behavior**

Distributed training to work without any issues or slowdowns.

**Environment overview (please complete the following information)**

 - Environment location: SLURM on AWS
 - Method of NeMo install: `pip install nemo_toolkit[asr]`

**Environment details**

- OS version: `Deep Learning Base GPU AMI (Ubuntu 20.04) ` (AWS)
- PyTorch version: `2.0.1+cu117`
- Python version: `3.8.10`

**Additional context**

GPU model: 8 A100