bug in multi-GPU fine tuning 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Nemo version V1.21.0,   script   ../examples/asr/speech_to_text_finetune.py 
ASR model fine tuning script runs fine with a single GPU, but fails for 2 GPUs and 3+ GPUs.

I made small changes to make multiple GPU training working, but want to confirm with the team if there is a better solution and maybe other cases that I should consider.

**Steps/Code to reproduce bug**

 python ../examples/asr/speech_to_text_finetune.py


**Expected behavior**
 
1. when trying to train with 2 GPUs, existing script gives the following error and training crash
File "*/.conda/envs/nemo_j/lib/python3.10/pathlib.py", line 1175, in mkdir self._accessor.mkdir(self, mode) FileExistsError: [Errno 17] File exists: '/model/save/path/~/checkpoints'
     
   This bug is fixed by making change in ".conda/envs/nemo_j/lib/python3.10/site-packages/nemo/core/classes/modelPT.py", line 368
         path.parent.mkdir(parents=True) -> path.parent.mkdir(parents=True,exist_ok=True))


2. when trying to train with 3+ GPUs, existing script gives the following error and training crash
   
      First, it gives a similar error to 2 GPU training,  but about a different folder.

             File "*/.conda/envs/nemo_j/lib/python3.10/pathlib.py", line 1175, in mkdir self._accessor.mkdir(self, mode) 
             FileExistsError: [Errno 17] File exists: '/model/save/path/~/run_0'

      This bug can be fixed similarly, by making a change in "*/.conda/envs/nemo_j/lib/python3.10/site-packages/nemo/utils/exp_manager.py", line 647
                
        new_run_dir.mkdir()-> new_run_dir.mkdir(exist_ok=True)
          
      Secondly, it gives an error about files missing:
             FileNotFoundError: [Errno 2] No such file or directory: '/model/save/path/nemo_log_globalrank-0_localrank-0.txt' -> '/model/save/path/run_0/nemo_log_globalrank-0_localrank-0.txt'

           It seems trying to move a file named nemo_log_globalrank-0_localrank-0.txt to a subfolder ./run_[0-9], which seems was moved already, so a simple fix was making a change in "*/.conda/envs/nemo_j/lib/python3.10/shutil.py", line 816

            os.rename(src, real_dst) -> pass

After these two changes, training with 3 and 4 GPUs scripts runs and WER reduces. 



**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 
        conda env installed on local Linux machine by the following commands:

               conda create --name nemo python==3.10.12 
               conda activate nemo 
               conda install -y pytorch=2.1.0 torchvision=0.16.0 torchaudio=2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia 
               pip install Cython==3.0.5 
               pip install nemo-toolkit['all']==1.21.0

**Environment details**
 

**Additional context**

     Training on single node with GPU A100.  
     My fix seems made the fine tuning working on 1 node with 2+ GPUs, not sure if this fix will generalize to multi-node, multi-GPU training. 

I've had the same issue

I looked into this a bit more and I think the real problem is that every process is getting mistaken as the master process (i.e. the issue is_global_rank_zero() from [get_rank.py ](https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/get_rank.py)). 

If I understand things correctly, [speech_to_text_finetune.py](https://github.com/NVIDIA/NeMo/blob/r1.21.0/examples/asr/speech_to_text_finetune.py) no longer uses torch.distributed.launch and instead uses the pytorch lightning Trainer. The problem is that is_global_rank_zero() checks for the environment variable `RANK` which is normally set by torch.distributed.launch but the pl Trainer that speech_to_text_finetune.py uses sets `LOCAL_RANK` for anything that is not the master process.

Long story short, the first check in is_global_rank_zero() returned None and always defaulted to checking SLURM_PROCID which was always 0 since I wasn't submitting an array job. All the processes thought they were the master process and got stuck in a loop trying to generate logs and backup logs in run_[0-3], directories.

Of course, i'm submitting a shell script with sbatch that includes calling speech_to_text_finetune.py and i'm wondering if i should just use sbatch to call speech_to_text_finetune.py directly.

Can someone from NVIDIA, check if I'm correct about this?


In the end I think the fix might be as simple as changing
[this line in get_rank.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/get_rank.py#L25) from 

`rank = get_envint("RANK", None)` to 
`rank = get_envint("LOCAL_RANK", None)`



@Jesteinbe note that the check of `LOCAL_RANK` already happens in https://github.com/NVIDIA/NeMo/blob/0bb9e66a6d29b28e8831d1d1dd8a30310173ce46/nemo/utils/get_rank.py#L39

The issues you are seeing may very well be related to how you start training. It all seems that you do use SLURM, but you run either an interactive job, or do not rely on SLURM to create the processes. There are multiple approaches even when using SLURM and some may/will have limitations. Additionally it is also important when the distributed processing is setup (this is when the Lightning/PyTorch variables for process rank get initialised), which may be later in code.

If there are no specific reasons against it, the easiest solution for you may be to rely on SLURM to create the processes, i.e. in your sbatch script set the parameter
```
#SBATCH --tasks-per-node=<num_gpus_per_node>
```
and call your training script as
```
srun train.sh
```

Thanks for the suggestion @itzsimpl. As soon as i posted that last comment I realized the same thing, i.e. the check for LOCAL_RANK happens later. 

Unfortunately I tried your solution but I'm still hitting the same problem, i.e. the SLURM_PROCID is always 0 so `is_global_rank_zero()` always returns true for every process. I also tried this directly

Here's how I submit the job:
```
ngpus=4
ncpus=1
sbatch --gres=gpu:$ngpus --tasks-per-node=$ngpus --cpus-per-gpu=$ncpus \
    ./utils/miniconda-wrapper.sh --env-file env/env.nemo.sh \
      python $NEMO_ROOT/examples/asr/speech_to_text_finetune.py <opts>
```

the `miniconda-wrapper.sh` just sets some things in PATH and loads a conda environment and wouldn't impact anything from a slurm perspective. Do you have any suggestions?

It might be that our slurm configuration is problematic. I come from an SGE background so I use slurm basically the same way I used SGE and I'm definitely not a slurm expert. That said, I tried a simple script that I thought should show different SLURM_PROCID:
```
#!/bin/bash
#SBATCH --ntasks=8
srun printenv SLURM_PROCID
```
expecting that it would print out
```
0
1
2
3
4
5
6
7
```
but when I ran it I only get 
```
0
```
So long story short, maybe the problem was with my grid not spawning tasks correctly. I'll keep working on things from my end but let me know if you have any suggestions in the meantime.

Assuming your latest env printing script is named `test.sbatch` run it with `sbatch test.sbatch`. The output should be something like this
```
$ sbatch test.sbatch 
Submitted batch job 17482323

$ cat slurm-17482323.out 
2
5
7
0
1
3
4
6
```

I was definitely calling things incorrectly but I'm still seeing weird behavior. 

Originally i had a script `asr.sh` which went through tokenization, training (which called `speech_to_text_finetune.py`) and decoding. I submitted that script with something along the lines of 

`sbatch  --gres=gpu:$ngpus --cpus-per-gpu=$num_workers ./asr.sh`

Within that script I was just calling `speech_to_text_finetune.py <opts>` instead of `srun speech_to_text_finetune.py <opts>` which prevented some of those env variables from getting updated correctly for `is_global_rank_zero()` to work properly. Once I switched to using `srun speech_to_text_finetune.py` then things worked. 

However when I switched my sbatch command to the recommendations from @itzsimpl i.e. 

`sbatch  --gres=gpu:$ngpus --tasks-per-node=$ngpus --cpus-per-task=$num_workers ./asr.sh` then i get the same behavior I had before and the training just hangs.

Is there a way to make, `is_global_rank_zero()` less brittle and more agnostic to how the job is scheduled? 

On a related note, I'll likely have to port my work on to a system running SGE as well. Is there documentation regarding how to submit jobs correctly using qsub?

Nice discussion here... @itzsimpl not sure why you are seeing such issues with finetune.py. 

However those fixes shouldn;t affect previous runs. @ericharper FYI

@itzsimpl can you create a PR with your changes 

@nithinraok For the issue discussed here I haven't made any changes, I was just trying to provide instructions on how to use Slurm, as that seems to me the primary cause of the issue that @Jesteinbe is seeing. I must admit that I haven't personally tried the script in question, though. 

I have a PR related to checkpointing open (https://github.com/NVIDIA/NeMo/pull/8277), but I'd say it is not related to what the OP is experiencing.

FWW. If anyone finds it helpful, we have a short deep-dive introductory tutorial on Slurm use https://docs.rdc.si/FRIDA/slurm/#usage. There are many more out there.

Sorry, I meant to tag @yunbin 

Good to know. In practice, my "fix", although not optimum, keeps multi-GPU
training going and WER keeps dropping.


On Tue, Feb 13, 2024 at 12:04 PM Nithin Rao ***@***.***>
wrote:

> Sorry, I meant to tag @yunbin <https://github.com/yunbin>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/issues/8333#issuecomment-1942019374>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABHWWSYL4IIV42FJYWT7WA3YTOMKPAVCNFSM6AAAAABC2RQV4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBSGAYTSMZXGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


-- 
Yunbin


@itzsimpl - I think i got things working correctly on my end though i had one question about the log that gets created during training, `nemo_log_globalrank-0_localrank-0.txt`. Should there be one of these for every GPU used in training? In some of my attempts to figure out the multi-gpu issues, I would occasionally see that happening. Now, the code seems to be running on multiple gpus but I only get the one log, `nemo_log_globalrank-0_localrank-0.txt`. Is that expected behavior?

i also tried the code out on another grid that was running Grid Engine and couldn't get it to work there. I think it's a different, but related, problem. In the Grid Engine case, the grid has all the gpus in exclusive processing mode and I immediately get error messages indicating the devices i requested are busy or unavailable despite having reserved all the gpus on the node and confirmed nothing was running there. When I got the code running on my Slurm cluster, I noticed that gpu 0 always had extra processes on it (one per every other gpu). My best guess is that things are failing on the Grid Engine cluster because Lightning is trying to kick off those extra processes but the gpu's are set in exclusive mode so things just hang. I posted on this Lightning issue about it since other people seemed to have seen similar behavior:

https://github.com/Lightning-AI/pytorch-lightning/discussions/9864

This is probably a Lightning problem more than Nemo but do you have any suggestions for how to deal with this situation?

@itzsimpl - there's definitely some sort of race condition occurring even when I submit jobs correctly via sbatch and srun. It took me a while to pin down but I've noticed different behavior when I run on different types of GPUs. On A100's there didn't seem to be any problems but on A5000's or A6000's I would see things hang. I assume this is more likely due to my cluster configuration than an artifact of the GPU architectures but it demonstrated that the race condition can still occur. Moreover, I occasionally see backup log directories, i.e. `run_0, run_1, ...` get generated even when i'm starting a brand new experiment and no log files existed previously. 

After looking more closely at [exp_manager.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py) I saw that i could specify a few flags to try, specifically `log_global_rank_0_only` and `seconds_to_sleep`. If I set `log_global_rank_0_only` to True then training no longer hangs. Or, if increase `seconds_to_sleep`, which controls how long the rank 0 process has to get started, from the  default of 5 to 10 then things no longer hang either. 

Throughout testing these things though, I realized I should probably  have been seeing the logs `nemo_log_globalrank-0_localrank-0.txt`, `nemo_log_globalrank-1_localrank-1.txt`, ... getting generated (assuming I didn't set `log_global_rank_0_only` to True) but I've only ever seen `nemo_log_globalrank-0_localrank-0.txt` get created. I think this points to the fact that the ranks of each individual gpu aren't getting properly identified.

Any thoughts?

@itzsimpl @nithinraok To add to this, I realized the `nemo_log_globalrank-0_localrank-0.txt` logs weren't getting generated correctly because LOCAL_RANK isn't getting set as an environment variable for me. All of the GPUS were therefore writing to `nemo_log_globalrank-0_localrank-0.txt` rather than their own logs.

Looking at [exp_manager.py.](https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L324-L328), I see comments about how `trainer.global_rank` and `trainer.is_global_zero` don't get set until `trainer.fit()` is called but that doesn't seem to be accurate anymore. I'm using Lightning v2.0.7 and it appears that `trainer.local_rank`, `trainer.global_rank`, and `trainer.is_global_zero` all get set correctly upon initialization without having to call `trainer.fit`. When I changed those lines in exp_manager.py to:
```
local_rank = trainer.local_rank                                                                                     
global_rank = trainer.global_rank                                                                                   
logging.rank = global_rank 
```
then all the logs get generated. Is there any reason why you wouldn't update everything in that script (and anywhere else in the codebase) to use `trainer.local_rank`, `trainer.global_rank` and `trainer.is_global_zero`? I assume this will have to propagate through many other scripts in NeMo as well.

That, or you'll need to point users to a specific version of lightning when users install Nemo. Otherwise I think they'll all potentially have similar problems.


@athitten is working on the update to 2.1, could you weigh in above?

@Jesteinbe just out of curiosity, do you perhaps name your Slurm job `interactive` or `bash`? You can check the value of the environment variable `SLURM_JOB_NAME`. If you do [Lightning will bypass Slurm](https://github.com/Lightning-AI/pytorch-lightning/blob/48c39ce24f814609c02aad208bc8c2df27145beb/src/lightning/fabric/plugins/environments/slurm.py#L103-L113).

Do you run NeMo on bare-metal (Conda or Modules) or via containers (Singularity, Pyxis+Enroot, Podman, ...)? Note that [Pyxis+Enroot will inject the variables `RANK` and `LOCAL_RANK`](https://github.com/NVIDIA/pyxis/blob/06eb95370e38275ffd3d714d4ab38f810a595a18/pyxis_slurmstepd.c#L1123-L1126) for a better UX. For other settings you may need to set these variables prior to calling your training script, e.g.:
```bash
# bare-metal (variables not set)
$ srun --ntasks=3 bash -c 'env | grep -i RANK'
PMIX_RANK=1
PMIX_RANK=2
PMIX_RANK=0

# pyxis+enroot (variables set)
$ srun --ntasks=3 --container-image=./nemo.sqfs bash -c 'env | grep RANK'
LOCAL_RANK=2
PMIX_RANK=2
RANK=2
LOCAL_RANK=1
PMIX_RANK=1
RANK=1
LOCAL_RANK=0
PMIX_RANK=0
RANK=0

# bare-metal (variables set on call)
$ srun --ntasks=3 bash -c 'RANK=$SLURM_PROCID LOCAL_RANK=$SLURM_LOCALID env | grep RANK'
LOCAL_RANK=1
RANK=1
PMIX_RANK=1
LOCAL_RANK=0
RANK=0
PMIX_RANK=0
LOCAL_RANK=2
RANK=2
PMIX_RANK=2
```

In addition, from within an interactive Slurm job you can run a multi-gpu training by calling
```bash
torchrun --standalone --nnodes=1 --nproc-per-node=<num_gpus> <script.py>
```
In this case `torchrun` will start multiple processes and [set up the environment variables](https://pytorch.org/docs/stable/elastic/run.html#environment-variables) appropriately; see [Torchrun documentation](https://pytorch.org/docs/stable/elastic/run.html#usage) for more details.


@itzsimpl I'm running on bare metal using a conda environment. Here's the hierarchy of the scripts I made:

`run.sh` -> High level "push-button" script that submits my ASR "recipe" to the slurm cluster via `sbatch`
`asr.sh` -> The ASR recipe that runs through training the tokenizer, training the ASR model, decoding, and scoring. ASR training is done with the Nemo script [speech_to_text_finetune.py](https://github.com/NVIDIA/NeMo/blob/v1.22.0/examples/asr/speech_to_text_finetune.py)

```
# run.sh amounts to basically just an sbatch command in the form:
sbatch --gres=gpu:4 --tasks-per-node=4 --cpus-per-task=8 -o slurm-%j-recipe.out -e slurm-%j-recipe.err \                                                    
    ./asr.sh <opts> 
```

then in asr.sh I call `srun python $NEMO_ROOT/examples/asr/speech_to_text_finetune.py`. I'm printing `os.environ` from speech_to_text_finetune.py and seeing that the `SLURM_JOB_NAME` is just getting set as "asr.sh", not "bash" or "interactive". Furthermore, I don't see `PMIX_RANK` in the printout so that's not getting set at all.

If `PMIX_RANK` is set or not depends on how Slurm was setup and configured, and it is not important in your case. By default `SLURM_JOB_NAME` gets set by sbatch to the name of the called script, you can set it to your own value by passing `--name=<name>`. You will see this name when you check the queue status.  

Based on your description of your environment, it seems that the issue you're seeing is because you're missing the environment variables `RANK` and `LOCAL_RANK`. `RANK` here is the global process id over multiple nodes, and `LOCAL_RANK` the node local process id. These variables are pretty standard for all multi-process jobs in all HPC environments, and a lot of code does depend on them, so do NeMo and Lightning. They are a commodity to help hide the complexity of setting up a multi-processing job where individual processes need to communicate with each other. 

Based on the description of your scripts, you can change your srun call to
``` 
srun bash -c 'RANK=$SLURM_PROCID LOCAL_RANK=$SLURM_LOCALID python $NEMO_ROOT/examples/asr/speech_to_text_finetune.py'
```

The trick here is to make the srun call first setup the environment and then execute your python script. This assumes that your Slurm environment is setup correctly (i.e. `SLURM_PROCID` and `SLURM_LOCALID` are set). You can test this by running (the order will not necessarily be the same)
```bash
$ srun --ntasks-per-node=2 --nodes=2 bash -c 'echo "SLURM_NODEID=$SLURM_NODEID SLURM_LOCALID=$SLURM_LOCALID SLURM_PROCID=$SLURM_PROCID"'
SLURM_NODEID=1 SLURM_LOCALID=1 SLURM_PROCID=3
SLURM_NODEID=1 SLURM_LOCALID=0 SLURM_PROCID=2
SLURM_NODEID=0 SLURM_LOCALID=1 SLURM_PROCID=1
SLURM_NODEID=0 SLURM_LOCALID=0 SLURM_PROCID=0
```

Note that every srun call in your `asr.sh` will create 4 parallel processes due to the directive `--tasks-per-node=4` that you passed to sbatch. Therefore if your script does multiple steps (tokenisation, training, decoding, scoring) each as a separate srun, you may need to consider if any of these need to be limited to one task only. You can do so by adding `--ntasks=1 --ntasks-per-node=1` to the specific srun call.

Note also that even with these variables properly set, when multiple-processes are run race conditions can occur as the order of startup is not deterministic. This is especially true for all steps that occur prior to to the initialization of `torch.distributed`, hence the current need for `seconds_to_sleep` which delays the startup of non-rank-zero processes. However, this tunable is dependent on the local environment (node HW, storage type, networking, ...). The default is a very good starting point, but if your environment is especially slow, you may need to tune the delay. 


@itzsimpl - Sorry to take so long to get back to this but I tried out your solution and it seems to mostly work.

One minor change was that I was specifying options to speech_to_text_finetune.py so I escaped those environment variables and used " instead of ' so i could still pass other variables. i.e.

```
srun bash -c "RANK=\$SLURM_PROCID LOCAL_RANK=\$SLURM_LOCALID python $NEMO_ROOT/examples/asr/speech_to_text_finetune.py --config=$my_config .."
```

That said, the race condition still occurs even when I set `seconds_to_sleep` as high as 20. It doesn't seem to affect the training other than the fact that old logs sometimes get overwritten in `run_N` directories. The cluster I'm running on consists almost entirely of hardware < 2years old so I'd like to think there shouldn't be that much latency but maybe something weird is happening.

Personally, I think it would be cleaner to check local and global ranks through Lightning's API since that's handling training anyway. Checking environment variables seems much less ideal, especially since it appears to be so dependent on how the cluster is configured.

@itzsimpl - Related to multi-GPU training and cluster configuration, i have two other questions. 

The first question is related to the hardware I'm using to train on. I've noticed that things work pretty seamlessly on my A100 nodes but things hang if I run them on A5000's or A6000's. The jobs seem to start up correctly and all the `nemo_log_globalrank-N_localrank-N.txt` logs start to get written but then everything hangs. I ran `nvidia-smi` and see 100% GPU utilization for every GPU and I've let it sit for as long as an hour but nothing changes. Conversely, when I run on an A100 training starts within about 1 minute of the job submission. Do you have any ideas what might cause that behavior?

The second question is that I need to run this code on another cluster that is running Grid Engine rather than Slurm. Do you have any pointers for training in an environment like that?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.