[NMT] Loss value becomes nan during training
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I launched the training of a model for the Machine Translation task, using the default [aayn_base.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/machine_translation/conf/aayn_base.yaml) config file. Everything seemed to be proceeding correctly, but after 126,218 steps in the third epoch something went wrong: the validation loss suddenly dropped from 2.78 to nan (Screenshot n.1), then returned to a finite value after 126,238 steps (Screenshot n.2) and finally returned to nan again after 126,586 steps (Screenshot n.3) and remained that way until the beginning of the fourth epoch, when I manually stopped the training. 

**Screenshot n.1**
<img width="652" alt="loss_nan_pt1" src="https://user-images.githubusercontent.com/112694503/196426908-1ffd5b6c-ca1a-4ee5-aeaa-2dd5a8560010.png">
**Screenshot n.2**
<img width="652" alt="loss_nan_pt2" src="https://user-images.githubusercontent.com/112694503/196426935-b05b1552-0b4a-41db-b35e-afd3519b2878.png">
**Screenshot n.3**
<img width="652" alt="loss_nan_pt3" src="https://user-images.githubusercontent.com/112694503/196426942-e4a1722b-a121-4729-953a-54f063963800.png">

Another strange thing I noticed is that the model seems to have suffered a strong performance degradation between the second and third epochs: I used one of the .nemo saved during the second epoch to compute the performance on a test set and got a BLEU score of ~29; if I compute the BLEU score, on the same test set, of the 5 best checkpoints saved by NeMo, on the other hand, I get a BLEU score of ~21 in all cases. Four of these checkpoints are from the third epoch, only one is from the second epoch.

**Steps/Code to reproduce bug**

I launched the training with the following command:

`enc_dec_nmt.py --config-path=conf --config-name=aayn_base trainer.devices=[0,1,2,3,4,5,6,7] ~trainer.max_epochs +trainer.max_steps=1000000 +trainer.val_check_interval=5000 ~model.test_ds model.encoder_tokenizer.vocab_size=32000 model.decoder_tokenizer.vocab_size=32000 model.max_generation_delta=5 model.train_ds.use_tarred_dataset=true model.src_language=src model.tgt_language=tgt model.preproc_out_dir=preprocessing_output model.train_ds.lines_per_dataset_fragment=10000000 model.train_ds.num_batches_per_tarfile=200 model.train_ds.src_file_name=outputs/split/train.src model.train_ds.tgt_file_name=outputs/split/train.tgt model.validation_ds.src_file_name=outputs/split/val.src model.validation_ds.tgt_file_name=outputs/split/val.tgt model.train_ds.tokens_in_batch=10000 model.validation_ds.tokens_in_batch=8192 model.optim.sched.warmup_ratio=0.05 +exp_manager.create_checkpoint_callback=True +exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU +exp_manager.checkpoint_callback_params.mode=max +exp_manager.checkpoint_callback_params.save_top_k=5 +exp_manager.exp_dir=NemoMT-TransformerBase +exp_manager.checkpoint_callback_params.always_save_nemo=True `

**Expected behavior**

A "proper" training, with loss not suddenly switching to nan.

**Environment overview and details**

- Nemo installed from pip using version 1.11.0
- apex installed from source cloning the repo
- Clean conda environment
- OS version:  Ubuntu 20.04.3 LTS
- Pytorch version: 1.12.1
- Python version 3.8.0
- Cuda version 11.6

**Additional context**

For the training I used 8 NVIDIA GeForce RTX 3090.