Fast Conformer training from scratch doesn't converge (WER = 1 ) , but it converges on pretrained model
State: closed | Pull request: no
Labels: name: bug

name: stale

Hello! I have a problem when I train ASR models from scratch on my arabic dataset

when I use pretrained fast-conformer model, It works well and converges

![image](https://github.com/NVIDIA/NeMo/assets/153502163/160bc027-00d4-4522-8953-9e4bdeb32a50)

and these are predicted and refrence results

![image](https://github.com/NVIDIA/NeMo/assets/153502163/6a85e4c7-a8d1-46c7-8929-4c11c76f1b37)

These are good results, reached 0.2 WER after about 10 epochs

but when I train the same model from scratch using config file on nemo github config files, It doesn't converges and these are results

![image](https://github.com/NVIDIA/NeMo/assets/153502163/c507f79f-7e3d-4cdf-9b92-d3a2bdd5f3a9)
![image](https://github.com/NVIDIA/NeMo/assets/153502163/28495f6b-167a-499b-a8c2-e4af1b461126)

I used command line and nemo notebooks, I used many ways for traning, but I always get the same results
I also tried to train the model from scratch using pretrained config file, but I have got the same results
What is the problem ?

What is the amount of data you are pretraining with ? We pretrain these asr models with 25,000 hours of speech. Small amounts of data for pretraining will mean you need to experiment a lot with hyper parameters to get it to converge. 

I am training with 150 hours of data, I trained it about 15 epochs ,
But WER is still 1 for both training and validation data, even the loss is still constant, the model does not learn even a little.

I observe similar effect, see this [discussion](https://github.com/NVIDIA/NeMo/discussions/7797). Still no clear answer why fastconformer doesn't converge from scratch. 

150 hours is enough to finetune but quite low to train from scratch. Still, it should train at least even if it gets poor wer. 

Can you print your entire model.optim config, plus mention what dropout and spec augment are you using ?

What is your lr, warmup steps and precision and also how many steps you have trained it for?

This is my model optim config

  optim:
    name: adamw
    lr: 1e-3
    # optimizer arguments
    betas: [0.9, 0.98]
    # less necessity for weight_decay as we already have large augmentations with SpecAug
    # you may need weight_decay for large models, stable AMP training, small datasets, or when lower augmentations are used
    # weight decay of 0.0 with lr of 2.0 also works fine
    weight_decay: 1e-3

    # scheduler setup
    sched:
      name: CosineAnnealing
      # scheduler config override
      warmup_steps: 15000
      warmup_ratio: null
      min_lr: 1e-4

dropout is 0.1
spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2 # set to zero to disable it
    # you may use lower time_masks for smaller models to have a faster convergence
    time_masks: 10 # set to zero to disable it
    freq_width: 27
    time_width: 0.05

I use the default configs for fast conformer config available here [](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/fastconformer/fast-conformer_ctc_bpe.yaml ), and I tried to use medium model but I get the same results

> I observe similar effect, see this [discussion](https://github.com/NVIDIA/NeMo/discussions/7797). Still no clear answer why fastconformer doesn't converge from scratch.

Thank you for this discussion, I checked it
I tried to use conformer, but I get similar results
please, tell me what was your dataset hours ? and how many epochs you trained to start getting results and WER started to decraese ?

> > I observe similar effect, see this [discussion](https://github.com/NVIDIA/NeMo/discussions/7797). Still no clear answer why fastconformer doesn't converge from scratch.
> 
> Thank you for this discussion, I checked it I tried to use conformer, but I get similar results please, tell me what was your dataset hours ? and how many epochs you trained to start getting results and WER started to decraese ?

@Dreahim , I experimented with a micro dataset of 900 examples (less than 2 hours), and my validation data had intersection with train. Batch size was 16 and no more than 400 epochs. Convergence - if it happens - begins before 50 epochs, sometimes earlier.

Is it a hybrid or rnnt model? How many steps you have trained for already?

> Is it a hybrid or rnnt model? How many steps you have trained for already?

It is fast conformer ctc, I trained for 2-3 epechs, and I stop it because there is no any convergion

I moved to conformer ctc, and there is no any convertion\
I tried a lot but I don't get any results , what is the problem ??!!

these are my configrations


```
name: "Conformer-CTC-BPE"

model:
  sample_rate: 16000
  log_prediction: true # enables logging sample predictions in the output during training
  ctc_reduction: 'mean_batch'
  skip_nan_grad: false

  train_ds:
    manifest_filepath: ???
    sample_rate: ${model.sample_rate}
    batch_size: 4 # you may increase batch_size if your memory allows
    shuffle: true
    num_workers: 8
    pin_memory: true
    max_duration: 50.0 # it is set for LibriSpeech, you may need to update it for your dataset
    min_duration: 0.1
    # tarred datasets
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    # bucketing params
    bucketing_strategy: "synced_randomized"
    bucketing_batch_size: null

  validation_ds:
    manifest_filepath: ???
    sample_rate: ${model.sample_rate}
    batch_size: 4 # you may increase batch_size if your memory allows
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true

  test_ds:
    manifest_filepath: null
    sample_rate: ${model.sample_rate}
    batch_size: 16 # you may increase batch_size if your memory allows
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true

  # recommend to SPE Unigram tokenizer with small vocab size of 128 or 256 when using 4x sub-sampling
  # you may find more detail on how to train a tokenizer at: /scripts/tokenizers/process_asr_text_tokenizer.py
  tokenizer:
    dir: ???  # path to directory which contains either tokenizer.model (bpe) or vocab.txt (wpe)
    type: bpe  # Can be either bpe (SentencePiece tokenizer) or wpe (WordPiece tokenizer)

  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: ${model.sample_rate}
    normalize: "per_feature"
    window_size: 0.025
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    log: true
    frame_splicing: 1
    dither: 0.00001
    pad_to: 0
    pad_value: 0.0

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2 # set to zero to disable it
    # you may use lower time_masks for smaller models to have a faster convergence
    time_masks: 5 # set to zero to disable it
    freq_width: 27
    time_width: 0.05

  encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: ${model.preprocessor.features}
    feat_out: -1 # you may set it if you need different output size other than the default d_model
    n_layers: 18
    d_model: 256

    # Sub-sampling params
    subsampling: striding # vggnet, striding, stacking or stacking_norm, dw_striding
    subsampling_factor: 4 # must be power of 2 for striding and vggnet
    subsampling_conv_channels: -1 # -1 sets it to d_model
    causal_downsampling: false

    # Feed forward module's params
    ff_expansion_factor: 4

    # Multi-headed Attention Module's params
    self_attention_model: rel_pos # rel_pos or abs_pos
    n_heads: 4 # may need to be lower for smaller d_models
    # [left, right] specifies the number of steps to be seen from left and right of each step in self-attention
    att_context_size: [-1, -1] # -1 means unlimited context
    att_context_style: regular # regular or chunked_limited
    xscaling: true # scales up the input embeddings by sqrt(d_model)
    untie_biases: true # unties the biases of the TransformerXL layers
    pos_emb_max_len: 5000

    # Convolution module's params
    conv_kernel_size: 31
    conv_norm_type: 'batch_norm' # batch_norm or layer_norm or groupnormN (N specifies the number of groups)
    # conv_context_size can be"causal" or a list of two integers while conv_context_size[0]+conv_context_size[1]+1==conv_kernel_size
    # null means [(kernel_size-1)//2, (kernel_size-1)//2], and 'causal' means [(kernel_size-1), 0]
    conv_context_size: null

    ### regularization
    dropout: 0.1 # The dropout used in most of the Conformer Modules
    dropout_pre_encoder: 0.1 # The dropout used before the encoder
    dropout_emb: 0.0 # The dropout used for embeddings
    dropout_att: 0.1 # The dropout for multi-headed attention modules

    # set to non-zero to enable stochastic depth
    stochastic_depth_drop_prob: 0.0
    stochastic_depth_mode: linear  # linear or uniform
    stochastic_depth_start_layer: 1

  decoder:
    _target_: nemo.collections.asr.modules.ConvASRDecoder
    feat_in: null
    num_classes: -1
    vocabulary: []

  # config for InterCTC loss: https://arxiv.org/abs/2102.03216
  # specify loss weights and which layers to use for InterCTC
  # e.g., to reproduce the paper results, set loss_weights: [0.3]
  # and apply_at_layers: [8] (assuming 18 layers). Note that final
  # layer loss coefficient is automatically adjusted (to 0.7 in above example)
  interctc:
    loss_weights: [0.3]
    apply_at_layers: [8]

  optim:
    name: adamw
    lr: 5.0
    # optimizer arguments
    betas: [0.9, 0.98]
    # less necessity for weight_decay as we already have large augmentations with SpecAug
    # you may need weight_decay for large models, stable AMP training, small datasets, or when lower augmentations are used
    # weight decay of 0.0 with lr of 2.0 also works fine
    weight_decay: 1e-3

    # scheduler setup
    sched:
      name: NoamAnnealing
      d_model: ${model.encoder.d_model}
      # scheduler config override
      warmup_steps: 10000
      warmup_ratio: null
      min_lr: 1e-6

trainer:
  devices: -1 # number of GPUs, -1 would use all available GPUs
  num_nodes: 1
  max_epochs: 1000
  max_steps: -1 # computed at runtime if not set
  val_check_interval: 1.0 # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
  accelerator: auto
  strategy: ddp
  accumulate_grad_batches: 1
  gradient_clip_val: 0.0
  precision: 32 # 16, 32, or bf16
  log_every_n_steps: 10  # Interval of logging.
  enable_progress_bar: True
  num_sanity_val_steps: 0 # number of steps to perform validation steps for sanity check the validation process before starting the training, setting to 0 disables it
  check_val_every_n_epoch: 1 # number of evaluations on validation every n epochs
  sync_batchnorm: true
  enable_checkpointing: False  # Provided by exp_manager
  logger: false  # Provided by exp_manager
  benchmark: false # needs to be false for models with variable-length speech input as it slows down training

exp_manager:
  exp_dir: null
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    # in case of multiple validation sets, first one is used
    monitor: "val_wer"
    mode: "min"
    save_top_k: 5
    always_save_nemo: True # saves the checkpoints as nemo files instead of PTL checkpoints

  # you need to set these two to True to continue the training
  resume_if_exists: false
  resume_ignore_no_checkpoint: false

  # You may use this section to create a W&B logger
  create_wandb_logger: false
  wandb_logger_kwargs:
    name: null
    project: null
```

Few things. Reduce spec augment time masks from 10 to 0 to see if model is learning at all. 

Lr lower to 5e-4. 

How many steps does it show per epoch ? Instead of setting warmup steps to 15000, set warmup ratio to 0.1 or so. 

You might need multiple Epochs - what is your hardware configuration ? Ie how many gpus + how many nodes. You might have to use gradient accumulation to increase the effective batch size to 256 at least. 

Training from scratch on smaller datasets is an effort in hyper parameters optimization. You will see that when we train on AN4 in the tutorials, we chose very specific optimization parameters in the ASR with Transducers tutorial. 

NeMo recommendation is to always always use fine-tuning instead of training from scratch unless it's for an academic paper. We have almost never seen a case that from scratch does better than fine-tuning from a model trained on thousands of hours of speech 

Epoch does not mean that much. How many steps you have trained? Have you passed the first 10k steps of warmup already? 

What is your global batch size? You should have at least 256 global batch in some cases to make sure ctc converges.

Have you tried rnnt or hybrid models? 

 Lowering lr as @titu1994 suggested is an option if you have smaller batch sizes. Other option is using grad accumulation.

> Few things. Reduce spec augment time masks from 10 to 0 to see if model is learning at all.
> 
> Lr lower to 5e-4.
> 
> How many steps does it show per epoch ? Instead of setting warmup steps to 15000, set warmup ratio to 0.1 or so.
> 
> You might need multiple Epochs - what is your hardware configuration ? Ie how many gpus + how many nodes. You might have to use gradient accumulation to increase the effective batch size to 256 at least.
> 
> Training from scratch on smaller datasets is an effort in hyper parameters optimization. You will see that when we train on AN4 in the tutorials, we chose very specific optimization parameters in the ASR with Transducers tutorial.
> 
> NeMo recommendation is to always always use fine-tuning instead of training from scratch unless it's for an academic paper. We have almost never seen a case that from scratch does better than fine-tuning from a model trained on thousands of hours of speech

Ok, I am going to try theese configrations, my GPU is RTX 3070 Ti so I can't increase batch size up to 256, I use batch size 1 for training

> Epoch does not mean that much. How many steps you have trained? Have you passed the first 10k steps of warmup already?
> 
> What is your global batch size? You should have at least 256 global batch in some cases to make sure ctc converges.
> 
> Have you tried rnnt or hybrid models?
> 
> Lowering lr as @titu1994 suggested is an option if you have smaller batch sizes. Other option is using grad accumulation.

Yes, I trained many times, one time I reached to 700k steps, and no results
my batch size is 1 or 2 because my GPU is 2070 TI, I can't raise more than this

I really don't think you can train a model to convergence from scratch on batch size 1. Fine-tuning is still possible, but not training from scratch.

Your only option is to enable and use grad accumulation to make the batch larger but you need many days just to pass the warmup stage. You may reduce the warmup but not sure how much you can reduce it.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.