[Question] Loss explodes instantly when fine-tuning Conformer-CTC on custom dataset
State: closed | Pull request: no

Hello!

I'm new to ASR (AI in general) and I'm struggling to fine-tune the Conformer CTC (small) on the [UASpeech dataset](http://www.isle.illinois.edu/sst/data/UASpeech/).

If I try to fine-tune the model from the NeMo pre-trained checkpoint, the loss starts around 40 and jumps to 4.14e+03 in the next step then some value around e+05 and stays in that range. Val_wer is nearly a constant 100%. It occasionally predicts long sequences of repeated characters (e.g. `iiiiiiiiiiiiiiiiiiiiiiiiii`). I'm wondering if I'm making some mistake when I change the vocabulary. I use the `process_asr_text_tokenizer.py` script giving 455 as the vocab size (the number of unique words in the dataset) to create the vocabulary then pass it to `model.cfg["train_ds"]["manifest_filepath"]`. I also set

```
model.cfg["decoder"]["vocabulary"] = []
model.cfg["decoder"]["num_classes"] = 455
```

Here's my `model.cfg`:
``` 
{'sample_rate': 16000, 
'log_prediction': True, 
'ctc_reduction': 'mean_batch', 
'train_ds': 
	{'manifest_filepath': path_to_custom_train_set, 
	'sample_rate': 16000, 
	'batch_size': 16, 
	'shuffle': True, 
	'num_workers': 8, 
	'pin_memory': True, 
	'use_start_end_token': False, 
	'trim_silence': False, 
	'max_duration': 16.7, 
	'min_duration': 0.1, 
	'is_tarred': False, 
	'tarred_audio_filepaths': None, 
	'shuffle_n': 2048, 
	'bucketing_strategy': 'synced_randomized', 
	'bucketing_batch_size': None}, 
'validation_ds': 
	{'manifest_filepath': path_to_custom_dataset_val_set, 
	'sample_rate': 16000, 
	'batch_size': 16, 
	'shuffle': False, 
	'num_workers': 8, 
	'pin_memory': True, 
	'use_start_end_token': False}, 
'test_ds': 
	{'manifest_filepath': ['/manifests/librispeech/librivox-dev-other.json', 
	'/manifests/librispeech/librivox-dev-clean.json', 
	'/manifests/librispeech/librivox-test-other.json', 
	'/manifests/librispeech/librivox-test-clean.json'], 
	'sample_rate': 16000, 
	'batch_size': 32, 
	'shuffle': False, 
	'num_workers': 8, 
	'pin_memory': True, 
	'use_start_end_token': False, 
	'is_tarred': False, 
	'tarred_audio_filepaths': ''}, 
'tokenizer': 
	{'dir': path_to_custom_tokenizer_spe_bpe, 
	'type': 'bpe'}, 
'preprocessor': 
	{'_target_': 'nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor', 
	'sample_rate': 16000, 
	'normalize': 'per_feature', 
	'window_size': 0.025, 
	'window_stride': 0.01, 
	'window': 'hann', 
	'features': 80, 
	'n_fft': 512, 
	'log': True, 
	'frame_splicing': 1, 
	'dither': 1e-05, 
	'pad_to': 0, 
	'pad_value': 0.0}, 
'spec_augment': 
	{'_target_': 'nemo.collections.asr.modules.SpectrogramAugmentation', 
	'freq_masks': 2, 
	'time_masks': 2, 
	'freq_width': 27, 
	'time_width': 100}, 
'encoder': 
	{'_target_': 'nemo.collections.asr.modules.ConformerEncoder', 
	'feat_in': 80, 
	'feat_out': -1, 
	'n_layers': 16, 
	'd_model': 176, 
	'subsampling': 'striding', 
	'subsampling_factor': 4, 
	'subsampling_conv_channels': 176, 
	'ff_expansion_factor': 4, 
	'self_attention_model': 'rel_pos', 
	'n_heads': 4, 
	'att_context_size': [-1, -1], 
	'xscaling': True, 
	'untie_biases': True, 
	'pos_emb_max_len': 5000, 
	'conv_kernel_size': 31, 
	'dropout': 0.1, 
	'dropout_emb': 0.0, 
	'dropout_att': 0.1}, 
'decoder': 
	{'_target_': 'nemo.collections.asr.modules.ConvASRDecoder', 
	'feat_in': 176, 
	'num_classes': 455, 
	'vocabulary': []}, 
'optim': 
	{'name': 'adamw', 
	'lr': 5, 
	'betas': [0.9, 
	0.98], 
	'weight_decay': 0, 
	'sched': 
		{'name': 'NoamAnnealing', 
		'd_model': 176, 
		'warmup_steps': 10000, 
		'warmup_ratio': None, 
		'min_lr': 1e-06}
	}, 
'init_weights_from_model': '/exp_init//ngc_checkpoints/small/ls_d176_adamlr2_wd0_aug5x0.05_wu10k_stridc176_ctc_2k_wpe128_untied_newrel.nemo', 
'target': 'nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE'}
```

I'm thinking that there might be an issue with the `init_weights_from_model` having a different vocabulary size because when I try to perform evaluation on the checkpoint I fine-tuned (by calling `model = nemo_asr.models.EncDecCTCModelBPE.load_from_checkpoint(path_to_checkpoint)`), I get the following error where the mismatch corresponds with the old and new vocabulary sizes.

```
RuntimeError: Error(s) in loading state_dict for EncDecCTCModelBPE:
	size mismatch for decoder.decoder_layers.0.weight: copying a param with shape torch.Size([129, 176, 1]) from checkpoint, the shape in current model is torch.Size([456, 176, 1]).
	size mismatch for decoder.decoder_layers.0.bias: copying a param with shape torch.Size([129]) from checkpoint, the shape in current model is torch.Size([456]).
```


The loss explosion doesn't happen if I start training from scratch (using a config file). The loss starts around 400 but decreases rapidly to 11/12 within the first epoch then plateaus.  After 60 epochs, the predictions still do not seem to improve (mostly random single letters), but maybe it just needs more training? Val_wer fluctuates between 99.9 and 100%.

1.  Any idea as to what I'm doing wrong here? It's good to note that the dataset is of impaired speech but it doesn't seem to be responsible for the loss issue since it doesn't happen when the model is trained from scratch.
2. When you use the tokenizer script, is it typical to do it just for the train manifest or a union of both? I'm only doing it for the train set to have a good "in the wild" evaluation.
3. When you have a small dataset, do you typically still need 2 distinct val and test sets? I use the same one for evaluation since it doesn't affect training, and I've noticed that existing papers do not mention a third split.


Thank you so much in advance!