fastconformer collapse after some improvment, seems to be irrelevant on the amount of data. 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
I'm trying to train a fastconformer model from scratch, I did manage to overfit for a small dataset, but it diverges for the full dataset, when I try a smaller dataset (30%) of the large it also diverges in what seems to be the exact same time:

<img width="614" alt="image" src="https://github.com/NVIDIA/NeMo/assets/8983713/1f955fd1-95f8-4505-b3db-ef3e4870dc8b">
<img width="614" alt="image" src="https://github.com/NVIDIA/NeMo/assets/8983713/7c447a33-5a72-4c5f-9b4e-af502e071520">


I've tried to lower the learning rate, but then the model just does not improve at all, the amount of data is about 1k-2k hours. I've also tried with 32bit, it did not help. 
Happy to hear if you have any intuition of what could help.

**Steps/Code to reproduce bug**

Here is the config file:
```
    model:
      sample_rate: 16000
      log_prediction: true
      ctc_reduction: mean_volume
      skip_nan_grad: false
      train_ds:
        manifest_filepath: PATH
        sample_rate: ${model.sample_rate}
        batch_size: 15
        shuffle: true
        num_workers: 8
        pin_memory: true
        max_duration: 16.7
        min_duration: 0.1
        is_tarred: false
        tarred_audio_filepaths: null
        shuffle_n: 2048
        bucketing_strategy: fully_randomized
        bucketing_batch_size: null
        normalize_overtime: false
        mean_overtime: false
      validation_ds:
        ds_item:
        - name: SILENT_GIP_inset
          manifest_filepath:  PATH
          sample_rate: ${model.sample_rate}
          batch_size: ${model.train_ds.batch_size}
          use_start_end_token: false
          num_workers: 8
          pin_memory: true
          max_duration: 16.7
          min_duration: 0.1
  tokenizer:
    dir: /PATH/tokenizer_spe_bpe_v128
    type: bpe
  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: ${model.sample_rate}
    normalize: "per_feature"
    window_size: 0.025
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    log: true
    frame_splicing: 1
    dither: 0.00001
    pad_to: 0
    pad_value: 0.0

      encoder:
        _target_: nemo.collections.asr.modules.ConformerEncoder
        feat_in: ${model.preprocessor.embedding_dim}
        feat_out: -1
        n_layers: 18
        d_model: 512
        subsampling: dw_striding
        subsampling_factor: 4
        subsampling_conv_channels: 256
        causal_downsampling: false
        subsampling_input_channels: 1
        ff_expansion_factor: 4
        self_attention_model: rel_pos
        n_heads: 8
        att_context_size:
        - -1
        - -1
        att_context_style: regular
        xscaling: true
        untie_biases: true
        pos_emb_max_len: 5000
        conv_kernel_size: 9
        conv_norm_type: batch_norm
        conv_context_size: null
        dropout: 0.1
        dropout_pre_encoder: 0.1
        dropout_emb: 0.0
        dropout_att: 0.1
        stochastic_depth_drop_prob: 0.0
        stochastic_depth_mode: linear
        stochastic_depth_start_layer: 1
      decoder:
        _target_: nemo.collections.asr.modules.ConvASRDecoder
        feat_in: null
        num_classes: -1
        vocabulary: []
      interctc:
        loss_weights: []
        apply_at_layers: []
      optim:
        name: adamw
        lr: 0.0003
        betas:
        - 0.9
        - 0.999
        weight_decay: 0
        sched:
          name: CosineAnnealing
          warmup_steps: 15000
          warmup_ratio: null
          min_lr: 2.9999999999999997e-05

    trainer:
      devices:
      - 0
      - 1
      num_nodes: 1
      max_epochs: 40
      max_steps: -1
      val_check_interval: 1.0
      accelerator: gpu
      strategy: ddp
      accumulate_grad_batches: 18
      gradient_clip_val: 0.0
      precision: bf16
      log_every_n_steps: 10
      enable_progress_bar: true
      num_sanity_val_steps: 0
      check_val_every_n_epoch: 1
      sync_batchnorm: true
      enable_checkpointing: false
      logger: false
      benchmark: false
    exp_manager:
      exp_dir: /.../nemo_experiments
      name: ${name}
      create_tensorboard_logger: true
      create_checkpoint_callback: true
      checkpoint_callback_params:
        monitor: val_wer
        mode: min
        save_top_k: 5
        always_save_nemo: true
      resume_from_checkpoint: null
      resume_if_exists: false
      resume_ignore_no_checkpoint: false
      create_wandb_logger: true

```

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: `git clone & pip install requirements`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version:
Linux

- PyTorch version
```
In [3]: torch.__version__
Out[3]: '2.2.0'```
- Python version
`Python 3.10.12`

2x4090



What batch size are you using for training ? Can you try grad accumulation to make your global batch size at least 128 ?

Thanks @titu1994 , I am using:
accumulate_grad_batches=18
batch_size=15
2 GPUs.
 So if I understand correctly,  the global batch size = 18 * 15 * 2 = 540, right?

Yes that much should be enough. It seems you're training from scratch, can you use a higher learning rate of 0.001 and see if it's learning something useful? 

Both plots above are just learning blank tokens, nothing more 

> Yes that much should be enough. It seems you're training from scratch, can you use a higher learning rate of 0.001 and see if it's learning something useful?

Yeah, I train from scratch this is an essential for the project.
I'll try, I actually only tried to reduce the learning rate, not increase it.

> Both plots above are just learning blank tokens, nothing more

What do you mean, I do have some good predictions in the logs, maybe I am missing something:
```
[NeMo I 2024-02-22 00:35:33 wer:319] reference:i have a strong feeling about this meeting
[NeMo I 2024-02-22 00:35:33 wer:320] predicted:i have a strong feeling about this meeting

[NeMo I 2024-02-22 00:35:36 wer:319] reference:where can i find gifts for my daughter near by
[NeMo I 2024-02-22 00:35:36 wer:320] predicted:onhe day fi so duhter near by

[NeMo I 2024-02-22 00:35:41 wer:319] reference:come on mia follow me he said
[NeMo I 2024-02-22 00:35:41 wer:320] predicted:em ci said
```

Thanks a lot for your help @titu1994 ! I would appreciate any other ideas that you have.


Your training batch wer is ~ 1, that would mean your model is learning nothing yet other than blank tokens. The text here - is it from later in training ?

Thanks @titu1994 I do manage to get some better results, but after some training the model just collapses:
<img width="438" alt="image" src="https://github.com/NVIDIA/NeMo/assets/8983713/f0c20ab5-31d4-4958-ab67-ecc2c1360a26">

Funny thing is that it depends on luck, some time if I resume from a previous checkpoint the model sometimes manage to continue converging (and then maybe collapse again and I try to continue from checkpoint once more).
<img width="461" alt="image" src="https://github.com/NVIDIA/NeMo/assets/8983713/c9ce2239-7566-4ede-ae83-ba3f84f1b2c3">
anything that I can add to increase stability ?
Thanks

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.