ASR - WER not decreasing after certain point (Finetuning hybrid_cache_aware_streaming model)
State: closed | Pull request: no
Labels: name: stale

I am trying to finetune STT En FastConformer Hybrid Transducer-CTC Large Streaming Multi model.

### **Issue**: WER does not go below 22.5%. Need suggestion of how I can improve this or anything that I might be doing wrong. 

### **Details:**

```
python3 -u NeMo/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py \
    --config-path="../conf/fastconformer/hybrid_cache_aware_streaming" \
    --config-name="fastconformer_hybrid_transducer_ctc_bpe_streaming" \
    +init_from_pretrained_model="stt_en_fastconformer_hybrid_large_streaming_multi" \
    model.train_ds.manifest_filepath="data/train_20.json" \
    model.validation_ds.manifest_filepath="data/val_20.json" \
    model.tokenizer.dir="default_tokenizer" \
    model.tokenizer.type="bpe" \
    trainer.max_epochs=300 \
    model.optim.name="adamw" \
    model.optim.weight_decay=0.0001 \
    model.optim.sched.warmup_steps=2000 \
    model.aux_ctc.ctc_loss_weight=0.3 \
    model.optim.lr=0.005 \
    model.optim.betas=[0.9,0.999] \
    ++exp_manager.exp_dir="checkpoints" \
    ++exp_manager.version=one \
    ++exp_manager.use_datetime_version=False \
    ++exp_manager.resume_ignore_no_checkpoint=True

```

### **Config:**

```
cfg:
  sample_rate: 16000
  compute_eval_loss: false
  log_prediction: true
  skip_nan_grad: false
  model_defaults:
    enc_hidden: 512
    pred_hidden: 640
    joint_hidden: 640
  train_ds:
    manifest_filepath: data/train_20.json
    sample_rate: 16000
    batch_size: 8
    shuffle: true
    num_workers: 8
    pin_memory: true
    max_duration: 20
    min_duration: 5
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    bucketing_strategy: synced_randomized
    bucketing_batch_size: null
  validation_ds:
    manifest_filepath: data/val_20.json
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true
  test_ds:
    manifest_filepath: null
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true
  tokenizer:
    dir: default_tokenizer
    type: bpe
  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: 16000
    normalize: NA
    window_size: 0.025
    window_stride: 0.01
    window: hann
    features: 80
    n_fft: 512
    frame_splicing: 1
    dither: 1.0e-05
    pad_to: 0
  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2
    time_masks: 10
    freq_width: 27
    time_width: 0.05
  encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: 80
    feat_out: -1
    n_layers: 17
    d_model: 512
    use_bias: true
    subsampling: dw_striding
    subsampling_factor: 8
    subsampling_conv_channels: 256
    causal_downsampling: true
    ff_expansion_factor: 4
    self_attention_model: rel_pos
    n_heads: 8
    att_context_size:
    - 70
    - 1
    att_context_style: chunked_limited
    att_context_probs: null
    xscaling: true
    pos_emb_max_len: 5000
    conv_kernel_size: 9
    conv_norm_type: layer_norm
    conv_context_size: causal
    dropout: 0.1
    dropout_pre_encoder: 0.1
    dropout_emb: 0.0
    dropout_att: 0.1
    stochastic_depth_drop_prob: 0.0
    stochastic_depth_mode: linear
    stochastic_depth_start_layer: 1
  decoder:
    _target_: nemo.collections.asr.modules.RNNTDecoder
    normalization_mode: null
    random_state_sampling: false
    blank_as_pad: true
    prednet:
      pred_hidden: 640
      pred_rnn_layers: 1
      t_max: null
      dropout: 0.2
    vocab_size: 1024
  joint:
    _target_: nemo.collections.asr.modules.RNNTJoint
    log_softmax: null
    preserve_memory: false
    fuse_loss_wer: true
    fused_batch_size: 4
    jointnet:
      joint_hidden: 640
      activation: relu
      dropout: 0.2
      encoder_hidden: 512
      pred_hidden: 640
    num_classes: 1024
  decoding:
    strategy: greedy_batch
    greedy:
      max_symbols: 10
    beam:
      beam_size: 2
      return_best_hypothesis: false
      score_norm: true
      tsd_max_sym_exp: 50
      alsd_max_target_len: 2.0
  aux_ctc:
    ctc_loss_weight: 0.3
    use_cer: false
    ctc_reduction: mean_batch
    decoder:
      _target_: nemo.collections.asr.modules.ConvASRDecoder
      feat_in: null
      num_classes: 1024
    decoding:
      strategy: greedy
  interctc:
    loss_weights: []
    apply_at_layers: []
  loss:
    loss_name: default
    warprnnt_numba_kwargs:
      fastemit_lambda: 0.005
      clamp: -1.0
  optim:
    name: adamw
    lr: 0.005
    betas:
    - 0.9
    - 0.999
    weight_decay: 0.0001
    sched:
      name: NoamAnnealing
      d_model: 512
      warmup_steps: 2000
      warmup_ratio: null
      min_lr: 1.0e-06
    target: nemo.collections.asr.models.hybrid_rnnt_ctc_bpe_models.EncDecHybridRNNTCTCBPEModel
  nemo_version: 2.1.0rc0

```

### **Things changed from default config:**
batch size: 8
**att_context_size: [70,1]** because I need to use this model for real time transcription. Experimented with the Online_ASR_Microphone_Demo_Cache_Aware_Streaming.ipynb and found out that [70,1] is what gives best latency for my use case of real time transcription (lookead_size = 80ms and encoder_step_length = 80ms)
**learning_rate: Tried with different lr (0.005, 0.0005, 0.0001) but not difference.**
min_duration = 5s
max_duration = 20s

### **Tokenizer:**
Used default tokenizer, saved the default tokenizer from the model and then used it for finetuning. (1024 SPE Unigram)

### **Dataset:**
**train: 120hrs** of english audio (**doctor patient conversations (recordings of meetings), contains medical jargon and american accent**), audio files: 5s-20s
converted all transcripts to only contain lowercase english alphabets, space and apostrophe
**val: 15hrs** of audio in same format as above

**WER of Original Model on val_set = 41%
The WER reaches 22-25% in all runs within first 10-15 epochs and after that it remains almost constant. Max epochs tried is 90.**

**GPU**: V100 16GB

**Tensorboard Logs of one of the runs:**
![Image](https://github.com/user-attachments/assets/495173e7-2ad4-472d-a952-579ed0ee062c)
![Image](https://github.com/user-attachments/assets/e3f2231e-7a5f-415d-9137-c1c1487b7c7a)
![Image](https://github.com/user-attachments/assets/7c153cf5-36af-431d-b706-c2f4c2052bf7)
![Image](https://github.com/user-attachments/assets/e54a7950-619c-4c90-aa3e-67767505b1d7)
![Image](https://github.com/user-attachments/assets/1dcd6028-4c0a-495a-8b51-342514daf081)



### UPDATE (30/09/2024)

**Tried adding more data to the dataset. Whole dataset is shuffled and then train and val sets are created. Didn't make any difference and instead now I have a bigger lowest VAL_WER with the new daataset.**

TRAIN: 220 hrs (~100 hrs more from previous data)
VAL: 17 hrs
Audio Lengths: 1s - 20s (1s 2s 3s 4s ....... 18s 19s 20s)
learning rate: 0.001
Tried with Batch size: 8 (V100), 32 (A100)
Epochs: 45 (V100), 95 and 300 (A100)
**VAL_WER: ~23-24% in all 3 runs**

For some reason it looks like the learning rate becomes equal to minimum learning rate (1e-6) right from the start and then stays there. Not sure why. Also don't know why training loss graph looks like as shown below. 
Also I am not sure what we are training here? CTC or RNNT or both with the config I am using. (I use decoder_type = "rnnt" when doing inference using this model)

Find tensorboard logs attached below

![Image](https://github.com/user-attachments/assets/2aea644b-e4cb-4468-9953-9293632592a4)
![Image](https://github.com/user-attachments/assets/38073ead-2551-4f5f-bca6-37ae3e56b9f8)
![Image](https://github.com/user-attachments/assets/34621941-cb64-4645-b7ff-5b864a545523)
![Image](https://github.com/user-attachments/assets/7ec32266-0e4b-47e5-b8b1-df3c279796da)
![Image](https://github.com/user-attachments/assets/a66dc5c1-08c3-4dd4-bd73-fa4af5e0a1ff)
![Image](https://github.com/user-attachments/assets/85779b5e-fc96-4494-8557-672fc8f2c4f4)



### UPDATE (3/10/2024)

I read in few answers in the issues that when we use **NoamAnnealing** which is the default scheduler mentioned in config ([fastconformer_hybrid_transducer_ctc_bpe_streaming.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/fastconformer/hybrid_cache_aware_streaming/fastconformer_hybrid_transducer_ctc_bpe_streaming.yaml)), **the lr is a multiplier and hence should be 1/10th or 1/5th of original**

I tried with lr = 0.5 (1 GPU A100 batch_size=32), 1.0 (4 GPUs A100 batch_size=32)
Min WER = 22% (44 epochs 1 GPU, 175 epochs 4 GPUs) 
and it seems almost stuck and doesn't go below this wer with new learning rates also

Apart from this, also tried below changes which also didn't make any difference 
att_context_size = [70,6] instead of [70,1]
accumulate_grad_batches = 8 instead of 1

@nithinraok can you please have a look into this and provide any suggestions or any insights that you have based on the provided details. 
Thank You

Attaching the logs from latest runs
![Image](https://github.com/user-attachments/assets/0cf9ca04-943b-48e1-8e56-cf84d93cf326)
![Image](https://github.com/user-attachments/assets/03ec744a-a768-48fc-a474-187124727054)
![Image](https://github.com/user-attachments/assets/e920cddc-d793-4e75-b5d4-dfce370b47bc)
![Image](https://github.com/user-attachments/assets/aab08b27-2e4a-4ece-bfa1-2a7582a5c9cd)
![Image](https://github.com/user-attachments/assets/5c1940a9-b978-4998-8948-8987109e14dc)



@nithinraok @titu1994 @elliottnv Any help/suggestions would be appreciated. Thank You.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.