ASR checkpoint saving error in 1.0.0rc1
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Error during saving first checkpoint in 1.0.0rc1 and main branches 
 
``` Epoch 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 563/563 [02:55<00:00,  3.20it/s, loss=767, v_num=6-59Epoch 0, global step 6: val_wer reached 1.00113 (best 1.00113), saving model to "nemo_experiments/QuartzNet15x5/2021-04-26_11-06-59/checkpoints/QuartzNet15x5---val_wer=1.00-epoch=0.ckpt" as top 3      
Saving latest checkpoint...
Epoch 0, global step 5: val_wer reached 1.00113 (best 1.00113), saving model to "nemo_experiments/QuartzNet15x5/2021-04-26_11-06-59/checkpoints/QuartzNet15x5---val_wer=1.00-epoch=0.ckpt" as top 3
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 560, in train
    self.train_loop.run_training_epoch()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 562, in run_training_epoch
    self.trainer.run_evaluation()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 667, in run_evaluation
    self.evaluation_loop.on_evaluation_end()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 110, in on_evaluation_end
    self.trainer.call_hook('on_validation_end', *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 924, in call_hook
    trainer_hook(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_hook.py", line 177, in on_validation_end
    callback.on_validation_end(self, self.get_model())
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 203, in on_validation_end
    self.save_checkpoint(trainer, pl_module)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 250, in save_checkpoint
    self._save_top_k_checkpoints(trainer, pl_module, monitor_candidates)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 592, in _save_top_k_checkpoints
    self._update_best_and_save(current, epoch, step, trainer, pl_module, metrics)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 644, in _update_best_and_save
    self._save_model(filepath, trainer, pl_module)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 361, in _save_model
    self.save_function(filepath, self.save_weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/properties.py", line 257, in save_checkpoint
    self.checkpoint_connector.save_checkpoint(filepath, weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 392, in save_checkpoint
    checkpoint = self.dump_checkpoint(weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 277, in dump_checkpoint
    callback_states = self.trainer.on_save_checkpoint()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_hook.py", line 199, in on_save_checkpoint
    state = callback.on_save_checkpoint(self, self.get_model())
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py", line 39, in wrapped_fn
    return fn(*args, **kwargs)
TypeError: on_save_checkpoint() missing 1 required positional argument: 'checkpoint'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NeMo/examples/asr/speech_to_text.py", line 54, in main
    trainer.fit(asr_model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 509, in fit
    results = self.accelerator_backend.train()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 152, in train
    results = self.ddp_train(process_idx=self.task_idx, model=model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 301, in ddp_train
    results = self.train_or_test()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in train_or_test
    results = self.trainer.train()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 591, in train
    self.train_loop.on_train_end()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 162, in on_train_end
    self.check_checkpoint_callback(should_save=True, is_last=True)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 196, in check_checkpoint_callback
    callback.on_validation_end(self.trainer, model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 203, in on_validation_end
    self.save_checkpoint(trainer, pl_module)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 250, in save_checkpoint
    self._save_top_k_checkpoints(trainer, pl_module, monitor_candidates)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 592, in _save_top_k_checkpoints
    self._update_best_and_save(current, epoch, step, trainer, pl_module, metrics)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 644, in _update_best_and_save
    self._save_model(filepath, trainer, pl_module)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 361, in _save_model
    self.save_function(filepath, self.save_weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/properties.py", line 257, in save_checkpoint
    self.checkpoint_connector.save_checkpoint(filepath, weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 392, in save_checkpoint
    checkpoint = self.dump_checkpoint(weights_only)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 277, in dump_checkpoint
    callback_states = self.trainer.on_save_checkpoint()
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_hook.py", line 199, in on_save_checkpoint
    state = callback.on_save_checkpoint(self, self.get_model())
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py", line 39, in wrapped_fn
    return fn(*args, **kwargs)
TypeError: on_save_checkpoint() missing 1 required positional argument: 'checkpoint'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Epoch 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 563/563 [02:56<00:00,  3.19it/s, loss=767, v_num=6-59]
``` 

**Steps/Code to reproduce bug**

Code error in branches 1.0.0rc1 and main with docker container 1.0.0rc1:

python NeMo/examples/asr/speech_to_text.py

But it works in the branch r1.0.0b4 with docker container r1.0.0b4

**Expected behavior**

Expected to save checkpoint.

**Environment overview (please complete the following information)**

docker run --runtime=nvidia -itd --ipc=host -p 8888:8888 -p 6006:6006 --restart unless-stopped --name nemo_100rc1 nvcr.io/nvidia/nemo:1.0.0rc1


**Additional context**

GPU model: v100


Please post your environment - torch version, pytorch lightning version, colab of local, etc. 

There's no official b4 docker container, so I assume a local docker container was built. 

Another thing is main branch codebase does not work with RC1 container, please build a container for main specifically to use that. 

I'm thinking it's an incompatible pytorch lightning version installed post installation of the RC1 container causing this issue 

Ok, I'll try to build a container for main.

I didn't build a local docker container.
I started to try from main branch on the latest nvcr.io/nvidia/nemo:1.0.0rc1 container. 

Ones It failed, I downgraded to branch 1.0.0rc1 and try with nvcr.io/nvidia/nemo:1.0.0rc1 container.

It again failed, I downgraded to branch r1.0.0b4 and try with nvcr.io/nvidia/nemo:r1.0.0b4 container. It works successfully. 

You can check versions of pytorch, pytorch lightning etc. in Dockerfile. 


Inside of the container, there is no need to install nemo again - and it is not adviced to do so since the container has a particular version of nemo with many dependencies. 

I was able to run the asr tutorial inside of the RC1 container without pip installing nemo.