Please document whether NEMO file contains best or last CKPT file
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

With the following YAML:

```
exp_manager:
  exp_dir: null
  name: ContextNet-8x-Stride-RNNT
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: val_wer
    mode: min
    save_top_k: 3
    always_save_nemo: true
  create_wandb_logger: false
  wandb_logger_kwargs:
    name: null
    project: null
    entity: null
  resume_if_exists: false
  resume_ignore_no_checkpoint: false
```
NeMo saves both .ckpt files and a .nemo file.

It is not 100% clear whether the best .CKPT file is what is used to save the .NEMO file, or whether the last .CKPT is used.

I'm guessing from a perusal of [exp_manager.py](https://github.com/NVIDIA/NeMo/blob/ddcc2a6c4eabc74b6b5021da05ff67d6d38ccf68/nemo/utils/exp_manager.py) that it saves the best model as the .NEMO file, but I can't say for sure by looking at the code.

[This document ](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/core.html#experiment-manager)is not clear and doesn't seem to mention all the keywords I see in the Python code.

**Describe the solution you'd like**

Please update the documentation to include all keywords related to exp_manager.py and which states clearly whether the best CKPT or last CKPT is saved in the training run's NEMO file.

**Hmmm**

I'm trying to verify this by comparing an element of the state dict of each of the CKPT files with the same element from the NEMO file and **none** of the CKPTs matches the NEMO file.  Is this to be expected?  So running

```
from glob import glob
ckpts=glob('./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/*.ckpt')
model.restore_from('./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/ContextNet-8x-Stride-RNNT.nemo')
sd1=model.state_dict();
for ckpt in ckpts:
    print("=====================================================\n\n\n")
    model2.load_from_checkpoint(ckpt);
    sd2=model2.state_dict();
    print(ckpt)
    print('NEMO')
    print(sd1['encoder.encoder.0.mconv.0.conv.weight'][0:3])
    print('CKPT')
    print(sd2['encoder.encoder.0.mconv.0.conv.weight'][0:3])
```

gives me (after deleting all the logging noise from NeMo):

```
./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/ContextNet-8x-Stride-RNNT--val_wer=0.1228-epoch=1.ckpt
NEMO
tensor([[[-0.3809, -0.0407,  0.5199,  0.0178, -0.8887]],

        [[-0.0082, -0.7741,  0.4612, -0.2517, -0.0893]],

        [[-0.2904, -0.7611, -0.0239,  0.3230,  0.4082]]])
CKPT
tensor([[[ 0.2958, -0.5322, -0.6329,  0.0123,  0.5627]],

        [[ 0.2680,  0.1954,  0.4858,  0.1461, -0.7119]],

        [[ 0.7032,  0.3157, -0.5824, -0.7284, -0.6969]]])
=====================================================
./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/ContextNet-8x-Stride-RNNT--val_wer=0.1735-epoch=19-last.ckpt
NEMO
tensor([[[-0.3809, -0.0407,  0.5199,  0.0178, -0.8887]],

        [[-0.0082, -0.7741,  0.4612, -0.2517, -0.0893]],

        [[-0.2904, -0.7611, -0.0239,  0.3230,  0.4082]]])
CKPT
tensor([[[ 0.2958, -0.5322, -0.6329,  0.0123,  0.5627]],

        [[ 0.2680,  0.1954,  0.4858,  0.1461, -0.7119]],

        [[ 0.7032,  0.3157, -0.5824, -0.7284, -0.6969]]])
=====================================================
./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/ContextNet-8x-Stride-RNNT--val_wer=0.1306-epoch=0.ckpt
NEMO
tensor([[[-0.3809, -0.0407,  0.5199,  0.0178, -0.8887]],

        [[-0.0082, -0.7741,  0.4612, -0.2517, -0.0893]],

        [[-0.2904, -0.7611, -0.0239,  0.3230,  0.4082]]])
CKPT
tensor([[[ 0.2958, -0.5322, -0.6329,  0.0123,  0.5627]],

        [[ 0.2680,  0.1954,  0.4858,  0.1461, -0.7119]],

        [[ 0.7032,  0.3157, -0.5824, -0.7284, -0.6969]]])
=====================================================
./nemo_experiments/ContextNet-8x-Stride-RNNT/2022-02-07_18-13-37/checkpoints/ContextNet-8x-Stride-RNNT--val_wer=0.1365-epoch=2.ckpt
NEMO
tensor([[[-0.3809, -0.0407,  0.5199,  0.0178, -0.8887]],

        [[-0.0082, -0.7741,  0.4612, -0.2517, -0.0893]],

        [[-0.2904, -0.7611, -0.0239,  0.3230,  0.4082]]])
CKPT
tensor([[[ 0.2958, -0.5322, -0.6329,  0.0123,  0.5627]],

        [[ 0.2680,  0.1954,  0.4858,  0.1461, -0.7119]],

        [[ 0.7032,  0.3157, -0.5824, -0.7284, -0.6969]]])

Also a little odd that all of the CKPTs for this element are the same.

**PPS**

If I just run EVAL but not TRAIN on the best CKPT and the NEMO file, they come out the same, so even the weights under the hood aren't the same, I guess they are the same model.  It would be nice to know why the weights above are different though.