Adding option to always create .nemo file when writing checkpoint
State: closed | Pull request: yes

Right now we only get a `.nemo` file when the training is finished.  However, it is common to let training continue for an unspecified amount of epochs and just stop when the training stops improving, in which case we'd never get a `.nemo` file.

This commit adds the option save the nemo file whenever a checkpoint is saved.

Use:

`+exp_manager.checkpoint_callback_params.always_save_nemo = True`

Signed-off-by: rprenger <rprenger@nvidia.com>

Could you make sure that at any point only 1 ".nemo" file is saved ? We dont want to clutter with dozens of nemo + dozens of PTL checkpoints for consecutive runs. 

> Could you make sure that at any point only 1 ".nemo" file is saved ? We dont want to clutter with dozens of nemo + dozens of PTL checkpoints for consecutive runs.

That's how it works now.  It'll either overwrite the old file with the latest or overwrite it with the best each time depending on your `save_best_model` option.  I basically copied the functionality of the `on_train_end` hook

I just realized that if "save_best_model" is True, this will make it so the training will continue after saving the checkpoint with the best model rather than the with the latest.  I'm making the PR a draft until I fix it.


Cool that sounds nice. Maybe a good default for this is True ? @blisc for final review 

also, @blisc please decide whether this OK for rc (looks small enough of a change to me) or should stay on main

Lightning's callbacks have `on_keyboard_interrupt`, https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html#on-keyboard-interrupt. Maybe we should just leverage that instead.

@blisc  this is for scenario when training takes several days, but you want to be able to grab intermediate results and evaluate them/deploy intermediate result to debugging tool, etc.

> Lightning's callbacks have `on_keyboard_interrupt`, https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html#on-keyboard-interrupt. Maybe we should just leverage that instead.

If we use the keyboard interrupt and we have a long running job, we can't have a .nemo file unless we kill our job.  We should be able to deploy our best model while it's still training.

> > Lightning's callbacks have `on_keyboard_interrupt`, https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html#on-keyboard-interrupt. Maybe we should just leverage that instead.
> 
> If we use the keyboard interrupt and we have a long running job, we can't have a .nemo file unless we kill our job. We should be able to deploy our best model while it's still training.

I'm confused as to why the lightning .ckpt isn't sufficient for your needs.

for translation, we have to use a model to produce segmented text before it's fed into the Nemo model (ie, sentencepiece processor). this must be stored alongside the model, but it is not itself part of the model we're training. a Nemo file contains this, but I dont think ckpt files would.  a model is not deployable without this preprocessing built-in.

furthermore, I believe that .ckpt files dont store the config that was used to create the model but that Nemo files do. that would be very useful to have to ensure reproducibility.