Strange convergence behavior of conformer transducer rnnt loss
State: closed | Pull request: no
Labels: name: stale

name: question

Is it ok that default model (I use ASR conformer transducer) has WER=0.26 but the same model that I have been training for 2 days has WER=0.32.
I use [Russian version](https://huggingface.co/nvidia/stt_ru_conformer_transducer_large) of the model.
My GPU is `V100DX-32Q`.

Here is the training config file:

```
cfg:
  sample_rate: 16000
  compute_eval_loss: true
  log_prediction: true
  skip_nan_grad: false
  model_defaults:
    enc_hidden: 512
    pred_hidden: 640
    joint_hidden: 640
  train_ds:
    manifest_filepath: data/processed/output_train_ds.json
    sample_rate: 16000
    batch_size: 32
    shuffle: true
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: false
    max_duration: 20.0
    min_duration: 0.1
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    bucketing_strategy: synced_randomized
    bucketing_batch_size: null
    bucketing_weights: ''
  validation_ds:
    manifest_filepath: data/processed/output_val_ds.json
    sample_rate: 16000
    batch_size: 32
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: false
  test_ds:
    manifest_filepath: null
    sample_rate: 16000
    batch_size: 16
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
  tokenizer:
    dir: /tokenizers/tokenizer_spe_unigram_v1024/
    type: bpe
    model_path: nemo:7e41df0fdb8f45388c8052fe688b7add_tokenizer.model
    vocab_path: nemo:531574790a7146c3ac33709c3b0e6aff_vocab.txt
    spe_tokenizer_vocab: nemo:8be1ca048a724049bfe9998715df9536_tokenizer.vocab
  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: 16000
    normalize: per_feature
    window_size: 0.025
    window_stride: 0.01
    window: hann
    features: 80
    n_fft: 512
    frame_splicing: 1
    dither: 1.0e-05
    pad_to: 0
  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2
    time_masks: 10
    freq_width: 27
    time_width: 0.05
  encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: 80
    feat_out: -1
    n_layers: 17
    d_model: 512
    subsampling: striding
    subsampling_factor: 4
    subsampling_conv_channels: 512
    causal_downsampling: false
    ff_expansion_factor: 4
    self_attention_model: rel_pos
    n_heads: 8
    att_context_size:
    - -1
    - -1
    att_context_style: regular
    xscaling: true
    untie_biases: true
    pos_emb_max_len: 5000
    conv_kernel_size: 31
    conv_norm_type: batch_norm
    conv_context_size: null
    dropout: 0.1
    dropout_emb: 0.0
    dropout_att: 0.1
  decoder:
    _target_: nemo.collections.asr.modules.RNNTDecoder
    normalization_mode: null
    random_state_sampling: false
    blank_as_pad: true
    prednet:
      pred_hidden: 640
      pred_rnn_layers: 1
      t_max: null
      dropout: 0.1
    vocab_size: 1024
  joint:
    _target_: nemo.collections.asr.modules.RNNTJoint
    log_softmax: null
    preserve_memory: false
    fuse_loss_wer: true
    fused_batch_size: 4
    jointnet:
      joint_hidden: 640
      activation: relu
      dropout: 0.1
      encoder_hidden: 512
      pred_hidden: 640
    num_classes: 1024
    vocabulary:
      # there are 1024 symbols so I hide them
  decoding:
    strategy: greedy_batch
    greedy:
      max_symbols: 10
    beam:
      beam_size: 2
      return_best_hypothesis: false
      score_norm: true
      tsd_max_sym_exp: 50
      alsd_max_target_len: 2.0
  loss:
    loss_name: default
    warprnnt_numba_kwargs:
      fastemit_lambda: 0.0
      clamp: -1.0
  variational_noise:
    start_step: 0
    std: 0.0
  optim:
    name: novograd
    lr: 0.0001
    betas:
    - 0.9
    - 0.98
    weight_decay: 0.001
    sched:
      name: CosineAnnealing
      warmup_steps: 10000
      warmup_ratio: null
      min_lr: 1.0e-06
  init_weights_from_model: /exp_init/ConformerL_rnnt_russian/Conformer_rnnt_trained_from_stt_en_ep_500_lr_2e-3_wd_1e-3_drc_cosine/Conformer_rnnt_trained_from_stt_en_ep_500_lr_2e-3_wd_1e-3_drc_cosine/checkpoints/Conformer_rnnt_en_finetuned_ep_500_lr_2e-3_wd_1e-3_drc_cosine.nemo
  labels:
    # there are 1024 labels so I hide them
  target: nemo.collections.asr.models.rnnt_bpe_models.EncDecRNNTBPEModel
  nemo_version: 1.12.0rc0
```
And I also set epochs_number to 1000 and accumulate_grad_batches to 8.


And for example this is val_wer for ctc (log_0410) and transducer (log_2410):
![image](https://github.com/NVIDIA/NeMo/assets/132354615/1e192c32-0c67-4ece-8e39-7a8b368fe35b)
val_wer converges almost linear

And here is val_loss:
![image](https://github.com/NVIDIA/NeMo/assets/132354615/27d8fd58-f818-4b3a-ae7d-844b7be5f153)