How to continue finetuning of .nemo model from 768th epoch?
State: closed | Pull request: no

I am using https://github.com/Open-Speech-EkStep/vakyansh-nemo-experimentation/blob/main/src/finetuning/speech_to_text_ctc_bpe.py to finetune. Due to power cut the process stopped after 768 epochs. So, I want to continue training the .nemo model from the 768th epoch. What extra arguments shall I pass 
to speech_to_text_ctc_bpe.py for that?
My .sh file is as follows:-
```
if [[ ${tokenizer_type}=bpe ]]; then

CUDA_VISIBLE_DEVICES=3,4 python /home/conformer/vakyansh-nemo-experimentation/src/finetuning/speech_to_text_ctc_bpe.py \
     --config-path=${config_path} \
     --config-name=${config_name} \
    model.train_ds.manifest_filepath=${train_manifest_path} \
    model.validation_ds.manifest_filepath=${valid_manifest_path} \
    model.test_ds.manifest_filepath=${test_manifest_path} \
    model.tokenizer.dir=${tokenizer_dir} \
    model.tokenizer.type=${tokenizer_type} \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.strategy="ddp" \
    exp_manager.resume_if_exists=true \
    exp_manager.resume_ignore_no_checkpoint=true \
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name=${wandb_run_name} \
    exp_manager.wandb_logger_kwargs.project=${wandb_project_name} \
    +init_from_pretrained_model=${pretrained_model}

fi
```
```[tasklist]
### Tasks
```


@okuchaiev a gentle reminder for the same

If you have the two resume flags set, simply calling the script multiple times with the same paths will be sufficient 

Yes, I have tried that but it is restarting from the 0th epoch.

Please check the folder called nemo_experiments and if it has a folder inside it which contains logs and a checkpoints dir, then add the following line to your training script 

exp_manager.explicit_log_dir=[prefix]/nemo_experiments/[name of the folder]

In the future you can add exp_manager.name="xyz" in order for NeMo to automatically detect the folder. 

Try
```
++exp_manager.resume_from_checkpoint="<path of your .nemo file>"
```

> exp_manager.resume_from_checkpoint
I have used the following .sh script :
```
if [[ ${tokenizer_type}=bpe ]]; then

CUDA_VISIBLE_DEVICES=3,4 python /home/conformer/vakyansh-nemo-experimentation/src/finetuning/speech_to_text_ctc_bpe.py \
     --config-path=${config_path} \
     --config-name=${config_name} \
    model.train_ds.manifest_filepath=${train_manifest_path} \
    model.validation_ds.manifest_filepath=${valid_manifest_path} \
    model.test_ds.manifest_filepath=${test_manifest_path} \
    model.tokenizer.dir=${tokenizer_dir} \
    model.tokenizer.type=${tokenizer_type} \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.strategy="ddp" \
    exp_manager.resume_if_exists=true \
    exp_manager.resume_ignore_no_checkpoint=true \
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name=${wandb_run_name} \
    exp_manager.wandb_logger_kwargs.project=${wandb_project_name} \
    +exp_manager.explicit_log_dir=${ckpt_dir} \
    +init_from_pretrained_model=${pretrained_model}

fi
```
But, it is showing the following error:-
```
Could not override 'exp_manager.explicit_log_dir'.
To append to your config use +exp_manager.explicit_log_dir=/home/conformer/nemo_experiments/Conformer-CTC-BPE/2023-11-22_17-23-06/
Key 'explicit_log_dir' is not in struct
    full_key: exp_manager.explicit_log_dir
    object_type=dict
```

You can add the key by using the provided instruction - 
 
To append to your config use ++exp_manager.explicit_log_dir=/home/conformer/nemo_experiments/Conformer-CTC-BPE/2023-11-22_17-23-06/


This worked for me

> You can add the key by using the provided instruction -
> 
> To append to your config use ++exp_manager.explicit_log_dir=/home/conformer/nemo_experiments/Conformer-CTC-BPE/2023-11-22_17-23-06/