Issue in FastSpeech2 | Restarting Training from a Saved Checkpoint.
State: closed | Pull request: no

I am experimenting with FastSpeech2 using LJSpeech dataset. I am trying to restart the run from a previously saved checkpoint, I tried changing the options in fastspeech2.yaml file. The log does say that it is resuming from earlier saved checkpoint, but the epoch value, train loss value, val loss value, all point to the fact that it's starting from scratch. How to resolve this? 

Below are the details:
_________________________________
**fastspeech2.yaml (showing relevant ones only)**
**trainer:**
  gpus: 2 # number of gpus
  max_epochs: 1000
  num_nodes: 1
**exp_manager:**
  exp_dir: "/home/NeMo/examples/tts/nemo_experiments"
  explicit_log_dir: null
  version: null
  name: ${name}
  create_tensorboard_logger: True
  create_checkpoint_callback: True
  resume_if_exists: True
  resume_ignore_no_checkpoint: True

**_(Tried all the permutations for exp_dir, explicit_exp_dir, version as well)_**
______________________________________
**Output Log (Showing relevant ones)**

[NeMo W 2021-11-16 18:03:07 exp_manager:545] No version folders would be created under the log folder as 'resume_if_exists' is enabled.
[NeMo I 2021-11-16 18:03:07 exp_manager:428] **Resuming from /home/NeMo/examples/tts/nemo_experiments/FastSpeech2/checkpoints/FastSpeech2--val_loss=3.9187-epoch=974-last.ckpt**
[NeMo I 2021-11-16 18:03:07 exp_manager:280] Experiments will be logged at /home/NeMo/examples/tts/nemo_experiments/FastSpeech2
[NeMo I 2021-11-16 18:03:07 exp_manager:635] TensorboardLogger has been set up
[NeMo W 2021-11-16 18:03:07 exp_manager:865] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to -1. Please ensure that max_steps will run for at least 25 epochs to ensure that checkpointing will not error out.

____________________________

**Training loss after resumption is an order or magnitude higher.**. For the 974th checkpoint, it was around 1.3e+03, but after resuming the training, it again jumps to 1.75e+04. I even tried to let the run go for some time, but every time found that the run starts from scratch (based on training/val losses and epoch number), though it says it is resuming from FastSpeech2--val_loss=3.9187-epoch=974-last.ckpt

grad graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Epoch 0:  32%|████████████████████████████████████████████████▎                                                                                                     | 68/211 [00:39<01:23,  1.72it/s, loss=Epoch 0:  33%|███████████████████████████████████████████▏                                                                                        | 69/211 [00:40<01:22,  1.72it/s, loss=1.81e+04, v_num=0]Epoch 1:  16%|████████████████████▋                                                                                                      
Epoch 1:  16%|█████████████████▋                                                                                            | 34/211 [00:19<01:42,  1.72it/s, loss=1.79e+04, v_num=0]Epoch 4:  49%|█████████████████████████████████████████████████████▋                    
Epoch 5:  94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉        | 198/211 [01:53<00:07,  1.74it/s, loss=1.76e+04, v_num=0]