Issue Resuming Training from Checkpoint with Small Validation Dataset
State: closed | Pull request: no
Labels: name: bug

name: stale

I encountered an issue while trying to resume training from a checkpoint in NeMo. When the validation dataset is less than twice the checkpoint step size, I get the following error:

```
File "/train/train_nemo_model.py", line 168, in train_nemo_model
    trainer.fit(model, ckpt_path=os.path.join(config["model"]["custom_model_save_path"], config["mlflow_cloud"]["run_id"]),
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1023, in _run_stage
    self.fit_loop.run()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 355, in advance
    self.epoch_loop.run(self._data_fetcher)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 134, in run
    self.on_advance_end()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 249, in on_advance_end
    self.val_loop.run()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 181, in _decorator
    return loop_run(self, *args, **kwargs)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 122, in run
    return self.on_run_end()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 244, in on_run_end
    self._on_evaluation_epoch_end()
File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 326, in _on_evaluation_epoch_end
    call._call_lightning_module_hook(trainer, hook_name)
File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 145, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
File "/nemo/core/classes/modelPT.py", line 917, in on_validation_epoch_end
    dataloader_logs = self.multi_validation_epoch_end(val_outputs, dataloader_idx=dataloader_idx)
File "/nemo/collections/asr/models/ctc_models.py", line 619, in multi_validation_epoch_end
    metrics = super().multi_validation_epoch_end(outputs, dataloader_idx)
File "/nemo/collections/asr/models/asr_model.py", line 36, in multi_validation_epoch_end
    if 'val_loss' in outputs[0]:
IndexError: list index out of range
```

**For example:**
The checkpoint step size is 28. I encounter the error when the dataloader processes 55 examples, but it works correctly with 57 examples. It seems that the issue arises when the validation dataset size is less than twice the checkpoint step size. Any advice on how to resolve this would be appreciated.



 - Environment location: local
 - Method of NeMo install: from source

**Environment details**

Otherwise, please provide:
- OS version - Ubuntu 22.04.4 LTS
- PyTorch version - pytorch-lightning==2.0.7
- Python version - python3.10

**Additional context**

GPU model - NVIDIA GeForce RTX 3060


I have the same issue,  it happens more when I use `use_semi_sorted_batching`
any help will be appreciated. Thanks

On Mon, May 27, 2024, 1:51 PM hayrapetyan8768 ***@***.***>
wrote:

> I encountered an issue while trying to resume training from a checkpoint
> in NeMo. When the validation dataset is less than twice the checkpoint step
> size, I get the following error:
>
> File "/train/train_nemo_model.py", line 168, in train_nemo_model
>     trainer.fit(model, ckpt_path=os.path.join(config["model"]["custom_model_save_path"], config["mlflow_cloud"]["run_id"]),
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
>     call._call_and_handle_interrupt(
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
>     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
>     return function(*args, **kwargs)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
>     self._run(model, ckpt_path=ckpt_path)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in _run
>     results = self._run_stage()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1023, in _run_stage
>     self.fit_loop.run()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
>     self.advance()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 355, in advance
>     self.epoch_loop.run(self._data_fetcher)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 134, in run
>     self.on_advance_end()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 249, in on_advance_end
>     self.val_loop.run()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 181, in _decorator
>     return loop_run(self, *args, **kwargs)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 122, in run
>     return self.on_run_end()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 244, in on_run_end
>     self._on_evaluation_epoch_end()
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 326, in _on_evaluation_epoch_end
>     call._call_lightning_module_hook(trainer, hook_name)
> File "/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 145, in _call_lightning_module_hook
>     output = fn(*args, **kwargs)
> File "/nemo/core/classes/modelPT.py", line 917, in on_validation_epoch_end
>     dataloader_logs = self.multi_validation_epoch_end(val_outputs, dataloader_idx=dataloader_idx)
> File "/nemo/collections/asr/models/ctc_models.py", line 619, in multi_validation_epoch_end
>     metrics = super().multi_validation_epoch_end(outputs, dataloader_idx)
> File "/nemo/collections/asr/models/asr_model.py", line 36, in multi_validation_epoch_end
>     if 'val_loss' in outputs[0]:
> IndexError: list index out of range
>
> *For example:*
> The checkpoint step size is 28. I encounter the error when the dataloader
> processes 55 examples, but it works correctly with 57 examples. It seems
> that the issue arises when the validation dataset size is less than twice
> the checkpoint step size. Any advice on how to resolve this would be
> appreciated.
>
>    - Environment location: local
>    - Method of NeMo install: from source
>
> *Environment details*
>
> Otherwise, please provide:
>
>    - OS version - Ubuntu 22.04.4 LTS
>    - PyTorch version - pytorch-lightning==2.0.7
>    - Python version - python3.10
>
> *Additional context*
>
> GPU model - NVIDIA GeForce RTX 3060
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/issues/9317>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACERJIJ5EYJJFJR427LGDITZEMFZTAVCNFSM6AAAAABIK6UGM6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYTQOBWGEZTQMA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@KunalDhawan @titu1994 @fedorovgv 
Might be relevant to your work, thanks

@hayrapetyan8768 @orena1 Can you please attach minimal reproducible code to reproduce the bug? 

It will be hard, it only happens sometimes, only on multi-GPUs examples, and only with big-datasets. 
Also, important to note, it only happens when we use mulitple validation datasets. 
https://github.com/NVIDIA/NeMo/pull/6306
Maybe also relevant for @bmwshop 

It is very strange that it is not deterministic and, if I understand correctly, it happens with and without ssb. Intuition suggests that this may not be a nemo, but pytorch lightning's >= 2.0 problem. Try to look in side like this https://github.com/Lightning-AI/pytorch-lightning/discussions/18110 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.