Model restored from checkpoint does not behave like model still in core
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Model restored from checkpoint does not behave like model still in core.

**Steps/Code to reproduce bug**

My model is checkpointed with the following callback:

```
model_save_dir='save/nemo_amharic'

class ModelCheckpointAtEpochEnd(pl.callbacks.ModelCheckpoint):
    def on_epoch_end(self, trainer, pl_module):
        metrics = trainer.callback_metrics
        metrics['epoch'] = trainer.current_epoch
        trainer.checkpoint_callback.on_validation_end(trainer, pl_module)

pid=os.getpid()
dt=datetime.datetime.now().strftime("%Y%m%d_%H%M%S")

checkpoint_callback = ModelCheckpointAtEpochEnd(
    filepath=model_save_dir+'/amharic_'+f'{dt}_{pid}'+'_{epoch:02d}',
    verbose=True,
    save_top_k=-1,
    save_weights_only=False,
    period=1)

trainer = pl.Trainer(gpus=[0], max_epochs=200, amp_level='O1', precision=16, checkpoint_callback=checkpoint_callback)
```

I train a model for a long time and then I run this check:

`pred=model.transcribe(paths2audio_files=audio_files, batch_size=1)`

I get back values that reflect training in particular:

```
gold	pred
0	አዎ እኔ አጋጣሚ አጭር ቀጠሮ ነበረችኝ እ	አዎ እኔ አጋጣሚ አጭር ቀጠሮ ነበረችኝ እ
1	አዎ ኑሯቸው ነዋ እንደ እኛ አቧራ እያቦነኑ አላደጉ	አዎ ኑሯቸው ነዋ እንደ እኛ አቧራ እያቦነኑ አላደጉ
2	ታድያ ምን ላደርግ ትምህርቱ እንደት ነው የማታው	ታድያ ምን ላደርግ ትምህርቱ እንደት ነው የማታው
3	ከሥራው ጋር ሄደልህ ግን	ሥራው ጋር ሄደልግህን
4	ከሥራው ጋር ሄደልህ	ከሥራው ጋር ሄደልህ
5	አይ አጣጥሞ መያዝ ነው በኋላ እሱ የሚያዋጣው ሲያልቅ	አይ አጣጥመ እኔው የያዝ ነው በኋላ እሱንን ያዋጣቴያልክ
6	አዲስ ሥራ ነው የገባኸው እንደ	አዲስ ሥራ ነው የገባኸው እንደ
7	የት ነው	የት ነው
8	እ	እ
9	ምን ጊዜያዊ ነው ኤርፖርት ወይስ ቋሚ	ምን ጊዜያዊ ነው ኤርፖርት ወይስ ቋሚ
10	ምን ሥራው አዎ አየር መንገድ እኮ ክፍያው ጥሩ እኮ ነው	ምን ሥራው አዎ አየር መንገድ እኮ ክፍያው ጥሩ እኮ ነው
11	አዎ ግን የፖኬት መኒ ምናምን የኪስ ገንዘብ ያስብሏቸዋል	አዎ ግን ያዎው ፐኬት መን ምናምን ኦኪከዛ አስቡላችኋም
```
I then reload the model from the training checkpoint (saved every epoch, and previous model stopped at end of epoch) as follows, and rerun the transcription on  the same files:

```
yaml2 = YAML(typ='safe')
with open('amharic_16000.yaml') as f:
    params2 = yaml2.load(f)
model2 = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
model2.change_vocabulary(new_vocabulary=params2['labels'])
model2.load_from_checkpoint('save/nemo_amharic/amharic_20201015_005720_353924_epoch=166.ckpt')
model2.cuda(0)
pred2=model2.transcribe(paths2audio_files=audio_files, batch_size=1)
pd.DataFrame(zip(transcripts,pred2), columns=['gold','pred'])
```
The transcription fails, it doesn't match the first transcription at all and looks like a totally untrained model:


```
gold	pred
0	አዎ እኔ አጋጣሚ አጭር ቀጠሮ ነበረችኝ እ	ኘፔኘፔሥኝጁኔጲኙሱድታጁገፌውሴፌሴፌሠኘኸዟሱእታገሪሱዌሡገኋዟቺጮዎገኸጴዟኳጆኳ...
1	አዎ ኑሯቸው ነዋ እንደ እኛ አቧራ እያቦነኑ አላደጉ	ፔኘኢኝሱኘገቺሱፌሴጁገፌሄሡጉጩገዙትቡገምጵሜኘቸሞሴጎሪፌፉሯቤሣትዙውዛዌገሯኳብ...
2	ታድያ ምን ላደርግ ትምህርቱ እንደት ነው የማታው	ኘግሲኘጁኘዛብኝጃዋጁዋገሉውድኋጁዟንትዟሞሴጁኝፎጋኤጴጲጿወዟለኘንዟዴሩኘሯኳጁኩ...
3	ከሥራው ጋር ሄደልህ ግን	ኘሥኘዟዋቺጃቺኘዋጴፌቺጴጆጲፌሴዳኘጫፌቤቺጽጲዉኳጫንቤሱዘኘገፌቤሱእኘጇፌቺሥኘሿ...
4	ከሥራው ጋር ሄደልህ	ኘሥሺዋጴሒሴማኸሿኘጴኒውጎሪጉገቱኘዛፌሉሱጆሠሿዋጎፌዙቺጉጮዮሠዋፌዟቭቸሻዋኸዟሴ...
5	አይ አጣጥሞ መያዝ ነው በኋላ እሱ የሚያዋጣው ሲያልቅ	ኘፔሥቺኝሁጁዛዟለፌቸፌገሪጵፌጫገዟሴሙዮጵጠጴፌውሴዲቸኝሯኽጮፌዙሱሩጆኘጴዟሴዟሣ...
6	አዲስ ሥራ ነው የገባኸው እንደ	ኘዟጁኘፔሥሿዣቺሥኘቺፔሥገዛቺውሱሿጫሸሉሱዉቤሥቾዟሴሏሣገኘዛቤለሷሸኘቤሯሥንዟሱ...
7	የት ነው	ኘሴሔሱጃጫሪዟሴታዌጢኸቤቺቄረዊኋሷጆሰቭኽኤኸሞቺሄጬለኘ
8	እ	ኘፔኘፔሴፔሴኘሥቺሥዋሁቺሱሿጁኸጁተጁሴኸታፌጴሥኢኘ
9	ምን ጊዜያዊ ነው ኤርፖርት ወይስ ቋሚ	ኘፔሥዟቺጓኘገዛቺውዚኘኸፌዟሌቺኘሁሱጴሌቺሱጴቹሱጫዋሁቺሱሁዋጫዋሔቤጴሱሌምሏኘጴ...
10	ምን ሥራው አዎ አየር መንገድ እኮ ክፍያው ጥሩ እኮ ነው	ኘዟኘዛዎሺዋሥፌዟውሴእኘታቤሥረጮቤኘትዟዋጮጆኤትዟቺጴፌጃገቾዟኘጴዟኘጴቱጁፌጃፌ...
11	አዎ ግን የፖኬት መኒ ምናምን የኪስ ገንዘብ ያስብሏቸዋል	ጴኘጓሥሺፔፌጼሥሱዉሥሴሥቺፔሥቺኘጁሥቺጃኬጁጃጁጧጁኘጲቺኘቢጁኘንዟዳጃዳጁዳሊዳጃ...

```

It seems like the weights are not getting loaded at all.  So I try a different save/restore route, outside of the checkpoint:

```
model.save_to('ihopethisworks.pt')
model2 = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
model2.change_vocabulary(new_vocabulary=params2['labels'])
model2.restore_from('ihopethisworks.pt')
model2.cuda(0)
pred2=model2.transcribe(paths2audio_files=audio_files, batch_size=1)
pd.DataFrame(zip(transcripts,pred2), columns=['gold','pred'])
```

This model also fails to reproduce the results of the model still in core, in the same way:

```
	gold	pred
0	አዎ እኔ አጋጣሚ አጭር ቀጠሮ ነበረችኝ እ	ሀፋሀሡቁሀዮፋማቹሀዮፋቢዮሾለቅጭፋማቪዶሬሀሡፋጾጹደጭፋቁዮሀሡፋኑጉፈቷማፋዮቷሀ...
1	አዎ ኑሯቸው ነዋ እንደ እኛ አቧራ እያቦነኑ አላደጉ	ሀቢቅቢቤሀፐቅቢማሀቅሀፐዮፋዦፈዓዮፋዮጴዥሀፈእፓጭፋሀዮፏፋጨቹዕሞለፋሕሙልሒቢጉ...
2	ታድያ ምን ላደርግ ትምህርቱ እንደት ነው የማታው	ሀፋፎፋፏሡኩቷኑቅሀቷፋቷቱቷዮጨኪጨፏታሀዮቷጪዎቷዮኙዮቷሬሀቷልፋቷጎሃፈጉሀሡቷሀ...
3	ከሥራው ጋር ሄደልህ ግን	ሀሡሀጥጾቱደቱሷሀደፋኔቱፋቤሀዮፋቷዮቱፋዥሀኩፋቷቱሷሀፋቷዌልሷቷፐልፋቷዥፁፑሀቷ...
4	ከሥራው ጋር ሄደልህ	ሀሡቤሬሀዮፋፓዮኖሀቷሙፋስፏሹፏፈቷሆቷሀቷልቢቷሆሀልዊጀዮሀቾልፋዮልፋዮልቷ ልፋ...
5	አይ አጣጥሞ መያዝ ነው በኋላ እሱ የሚያዋጣው ሲያልቅ	ሀሡቅቤሀቅሀዩለፏልሀጉቢልዮዥፐልፋማልሀሡኽሡጅሡፋሀልቢግኑሀቷፋዮሀፋሒኦሬሀ ዎ...
6	አዲስ ሥራ ነው የገባኸው እንደ	ሀእሡቷቱልቷሀዮልፋቷቤቅሀቷልፋቷዥሬሀዌልፋዮለፎዶሀፎሙፋፌልፋዥሀዮልፋዮቱፑሀዮ...
7	የት ነው	ሀፋሀሡቢቤልፋፏዌፑቷፋቢዮኧጤሀጤልፋቤባሀልፋሡማጉፀግ
8	እ	ሀፋሀዮፋቱሡቁቤሀዮፋቤሬሀሡፋፈጾቹፐግሣፋልማፀግ
9	ምን ጊዜያዊ ነው ኤርፖርት ወይስ ቋሚ	ሀፋሀፋሀሡፋኔቷቤሥፋቷጬሀፋቱደቱደሀቷሀፋቷታልፋቷሀልፋቷዮልሀዮፋማሀልሀቷሡፋቱ...
10	ምን ሥራው አዎ አየር መንገድ እኮ ክፍያው ጥሩ እኮ ነው	ሀሡቤሀዮፋኢዮጎቷልፋዮዌቾዥቷልዎቢዮጌፋቷዮጴፋሡፋቤሏሙቢቤሆሏዎቷሏፐዎቷሆሏቷፋ...
11	አዎ ግን የፖኬት መኒ ምናምን የኪስ ገንዘብ ያስብሏቸዋል	ሀፋእፋኔሷኔቷሷቷግቷሡግቷዮግፋጥሡቷቤጬፋኔፋኔቷኔፋቤቃግቷኔፋቷግቷፋቷፋቷፋቷል...
```

**Expected behavior**

Transcription of model in core and model recreated from in core model using both checkpoint/restore from checkpoint and save to/restore from methods should result in the same transcription.

**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 20.04, Anaconda Python 3.7
 - Method of NeMo install: pip install

**Environment details**

- OS version: Ubuntu 20.04LTS
- PyTorch version: 1.6.0
- Python version: 3.7.4
- GPU model: NVidia GeForce RTX 2080TI
- CUDA: 10.1.243
- cuDNN: 7.6.5

** YAML file**

The YAML file `amharic_16000.yaml` loaded above is as follows:

```
name: &name "QuartzNet15x5"
sample_rate: &sample_rate 16000
repeat: &repeat 1
dropout: &dropout 0.0
separable: &separable true
labels: &labels [' ', ሀ, ሁ, ሂ, ሃ, ሄ, ህ, ሆ, ለ, ሉ,
                      ሊ, ላ, ሌ, ል, ሎ, ሏ, ሐ, ሑ, ሒ, ሔ, ሕ, መ, ሙ,
                      ሚ, ማ, ሜ, ም, ሞ, ሟ, ሠ, ሡ, ሣ, ሥ, ሦ, ረ, ሩ,
                      ሪ, ራ, ሬ, ር, ሮ, ሯ, ሰ, ሱ, ሲ, ሳ, ሴ, ስ, ሶ,
                      ሷ, ሸ, ሹ, ሺ, ሻ, ሼ, ሽ, ሾ, ሿ, ቀ, ቁ, ቂ, ቃ,
                      ቄ, ቅ, ቆ, ቋ, በ, ቡ, ቢ, ባ, ቤ, ብ, ቦ, ቧ, ቨ,
                      ቪ, ቫ, ቬ, ቭ, ተ, ቱ, ቲ, ታ, ቴ, ት, ቶ, ቷ, ቸ,
                      ቹ, ቺ, ቻ, ቼ, ች, ቾ, ቿ, ኋ, ነ, ኑ, ኒ, ና, ኔ,
                      ን, ኖ, ኗ, ኘ, ኙ, ኚ, ኛ, ኜ, ኝ, ኞ, አ, ኡ, ኢ,
                      ኤ, እ, ኦ, ኧ, ከ, ኩ, ኪ, ካ, ኬ, ክ, ኮ, ኳ, ኸ,
                      ኻ, ኼ, ኽ, ወ, ዉ, ዊ, ዋ, ዌ, ው, ዎ, ዐ, ዑ, ዓ,
                      ዕ, ዘ, ዙ, ዚ, ዛ, ዜ, ዝ, ዞ, ዟ, ዠ, ዢ, ዣ, ዤ,
                      ዥ, ዦ, የ, ዩ, ዪ, ያ, ዬ, ይ, ዮ, ደ, ዱ, ዲ, ዳ,
                      ዴ, ድ, ዶ, ዷ, ጀ, ጁ, ጂ, ጃ, ጄ, ጅ, ጆ, ጇ, ገ,
                      ጉ, ጊ, ጋ, ጌ, ግ, ጎ, ጓ, ጠ, ጡ, ጢ, ጣ, ጤ, ጥ,
                      ጦ, ጧ, ጨ, ጩ, ጪ, ጫ, ጬ, ጭ, ጮ, ጱ, ጲ, ጳ, ጴ,
                      ጵ, ጸ, ጹ, ጺ, ጻ, ጼ, ጽ, ጾ, ጿ, ፀ, ፁ, ፃ, ፄ,
                      ፅ, ፆ, ፈ, ፉ, ፊ, ፋ, ፌ, ፍ, ፎ, ፏ, ፐ, ፑ, ፒ,
                      ፓ, ፔ, ፕ, ፖ]

model:
  train_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: *labels
    batch_size: 16
    trim_silence: True
    max_duration: 33
    shuffle: True
    is_tarred: False
    tarred_audio_filepaths: null

  validation_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: *labels
    batch_size: 16
    shuffle: False

  preprocessor:
    cls: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    params:
      normalize: "per_feature"
      window_size: 0.02
      sample_rate: *sample_rate
      window_stride: 0.01
      window: "hann"
      features: &n_mels 64
      n_fft: 512
      frame_splicing: 1
      dither: 0.00001
      stft_conv: false

  spec_augment:
    cls: nemo.collections.asr.modules.SpectrogramAugmentation
    params:
      rect_freq: 50
      rect_masks: 5
      rect_time: 120

  encoder:
    cls: nemo.collections.asr.modules.ConvASREncoder
    params:
      feat_in: *n_mels
      activation: relu
      conv_mask: true

      jasper:
        - filters: 128
          repeat: 1
          kernel: [11]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [13]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [15]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [17]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [19]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: 1
          kernel: [21]
          stride: [1]
          dilation: [1]
          dropout: 0.0
          residual: false
          separable: *separable
          se: true
          se_context_size: -1

        - filters: &enc_feat_out 1024
          repeat: 1
          kernel: [1]
          stride: [1]
          dilation: [1]
          dropout: 0.0
          residual: false
          separable: *separable
          se: true
          se_context_size: -1

  decoder:
    cls: nemo.collections.asr.modules.ConvASRDecoder
    params:
      feat_in: 1024
      num_classes: 28
      vocabulary: *labels

  optim:
    name: novograd
    # cls: nemo.core.optim.optimizers.Novograd
    lr: .01
    # optimizer arguments
    betas: [0.8, 0.5]
    weight_decay: 0.001

    # scheduler setup
    sched:
      name: CosineAnnealing

      # pytorch lightning args
      monitor: val_loss
      reduce_on_plateau: false

      # Scheduler params
      warmup_steps: null
      warmup_ratio: null
      min_lr: 0.0
      last_epoch: -1

trainer:
  gpus: 0 # number of gpus
  max_epochs: 5
  max_steps: null # computed at runtime if not set
  num_nodes: 1
  distributed_backend: ddp
  accumulate_grad_batches: 1
  checkpoint_callback: False  # Provided by exp_manager
  logger: False  # Provided by exp_manager
  row_log_interval: 1  # Interval of logging.
  val_check_interval: 1.0 # check once per epoch .25 for 4 times per epoch

exp_manager:
  exp_dir: null
  name: *name
  create_tensorboard_logger: True
  create_checkpoint_callback: True

hydra:
  run:
    dir: .
  job_logging:
    root:
      handlers: null

```

[This question ](https://github.com/NVIDIA/NeMo/issues/1202) has the answer.  Loading the pretrained model and then changing the vocabulary and reloading the weights doesn't work. There is apparently a lot more state there that doesn't get recovered that way.  It turns out there is a much simpler way to reload, that gives back the trained result of the in-core model:

`model2 = nemo_asr.models.EncDecCTCModel.restore_from('ihopethisworks.pt')`

rather than

```
model2 = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
model2.change_vocabulary(new_vocabulary=params2['labels'])
model2.restore_from('ihopethisworks.pt')
```

This also works, i.e. the alternative method using the checkpoint:

`model3 = nemo_asr.models.EncDecCTCModel.load_from_checkpoint('amharic_20201015_005720_353924_epoch=167.ckpt')
`

This is something that could be spelled out better in the ASR_with_NeMo.ipynb tutorial notebook.

yes, ```model2.load_from_checkpoint('save/nemo_amharic/amharic_20201015_005720_353924_epoch=166.ckpt')``` does not change ```model2``` object because it is a class method of Pytorch LightningModule:
https://github.com/PyTorchLightning/pytorch-lightning/blob/8f2bba10e198e42b9fcc3656719c571a404230a0/pytorch_lightning/core/saving.py#L53 

we are working to make saving/restoring less confusing in the next update