Cannot  resume  fro checkpoint
State: closed | Pull request: no
Labels: name: bug

name: stale

i am training  conformer ctc  so  i was in  epoch  3 epoch  three   ended  but  it didnt  save the  checkpoint for   that  epoch  rerunning  again it is not  getting  any checkpoint neither 2 or  one  how  can i resolve  this  it is starting again from  zero  it i saying no  checkpoint  found

<code>

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[NeMo W 2023-11-11 08:05:28 exp_manager:754] No version folders would be created under the log folder as 'resume_if_exists' is enabled.
Error executing job with overrides: ['exp_manager.name=SwahiliModelTraining', 'exp_manager.resume_if_exists=true', 'exp_manager.resume_ignore_no_checkpoint=false', 'exp_manager.exp_dir=drive/MyDrive/hackathon/', 'model.tokenizer.dir=drive/MyDrive/hackathon/tokenizer_spe_bpe_v1024_max_4', 'model.train_ds.is_tarred=true', 'model.train_ds.tarred_audio_filepaths=drive/MyDrive/hackathon/train_tarred_1bk/audio__OP_0..1023_CL_.tar', 'model.train_ds.manifest_filepath=drive/MyDrive/hackathon/train_tarred_1bk/tarred_audio_manifest.json', 'model.validation_ds.manifest_filepath=/content/drive/MyDrive/hackathon/eval_decoded_processed.json', 'model.test_ds.manifest_filepath=drive/MyDrive/hackathon/test_decoded_processed_v2.json']
Traceback (most recent call last):
  File "/content/drive/MyDrive/hackathon/scripts/speech_to_text_ctc_bpe.py", line 81, in main
    exp_manager(trainer, cfg.get("exp_manager", None))
  File "/usr/local/lib/python3.10/dist-packages/nemo/utils/exp_manager.py", line 356, in exp_manager
    check_resume(
  File "/usr/local/lib/python3.10/dist-packages/nemo/utils/exp_manager.py", line 613, in check_resume
    raise NotFoundError(
nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :drive/MyDrive/hackathon/SwahiliModelTraining/checkpoints. Cannot resume.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

</code>


as  you  can see  above  but  the  model  when i run interference  it is working  fine


In these cases, use a predefined version name for exp_manager, as you mentioned `exp_manager.resume_ignore_no_checkpoint=false` which means it would error out if there are no checkpoints in save directory. 
use `exp_manager.version=v1` or something like that.  

it  is  not  working  though


what do you mean by not working? Can you elaborate? have you changed to `exp_manager.resume_ignore_no_checkpoint=true` ?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

> it is not working though

Hi @nithinraok  did you figure this out. I just solved a similar. I copied the checkpoint folder and all other training artifacts to my training dir which is  `drive/MyDrive/hackathon/SwahiliModelTraining/` as in your case now.