Empty Prediction when training from scratch a Squeezeformer-CTC with custom tokenizer.
State: closed | Pull request: no

Hello everybody,

I'm trying to train an extra-small Squeezeformer-CTC ASR with parameters indicated in the basic [yaml configuration](https://github.com/NVIDIA/NeMo/blob/03a8f1560fb58b7a6b4749d078863b04c4a299af/examples/asr/conf/squeezeformer/squeezeformer_ctc_bpe.yaml). Since I'm using a custom dataset, I had to make a custom bpe tokenizer.  However, from the earliest epochs, predictions are empty, and the WER is constant at 1.0. Here is a sample prediction:

    [NeMo I 2022-11-09 19:21:52 wer_bpe:261] reference:se voi volete realizzare attraverso
    [NeMo I 2022-11-09 19:21:52 wer_bpe:262] predicted:

In the following you can see the configuration:

      exp_dir: new_model
      exp_manager:
        checkpoint_callback_params:
          always_save_nemo: true
          mode: min
          monitor: val_wer
          save_top_k: 5
        create_checkpoint_callback: true
        create_tensorboard_logger: true
        create_wandb_logger: false
        exp_dir: null
        name: ${name}
        resume_if_exists: false
        resume_ignore_no_checkpoint: false
        wandb_logger_kwargs:
          name: null
          project: null
      model:
        ctc_reduction: mean_batch
        decoder:
          _target_: nemo.collections.asr.modules.ConvASRDecoder
          feat_in: 196
          num_classes: 128
          vocabulary:
          - <unk>
          - ▁
          - a
          - o
          - s
          - n
          - u
          - re
          - i
          - r
          - e
          - t
          - l
          - g
          - ta
          - ▁di
          - m
          - ▁a
          - ▁e
          - te
          - ra
          - ri
          - 'no'
          - to
          - c
          - ti
          - p
          - le
          - la
          - li
          - ▁in
          - b
          - ci
          - ▁del
          - si
          - ni
          - ''''
          - ▁che
          - me
          - ▁s
          - so
          - ro
          - ▁un
          - f
          - ve
          - ca
          - ne
          - co
          - ▁co
          - ▁con
          - mi
          - do
          - ▁per
          - ▁la
          - di
          - d
          - zione
          - na
          - ▁al
          - ce
          - vi
          - ▁de
          - vo
          - ▁il
          - bi
          - se
          - ▁p
          - pe
          - lo
          - ▁ri
          - ▁pa
          - de
          - va
          - ma
          - ▁si
          - po
          - ▁i
          - ▁pro
          - ▁quest
          - ▁po
          - ▁è
          - nti
          - gli
          - ▁da
          - fi
          - tto
          - tà
          - tti
          - ▁l
          - ▁non
          - ▁su
          - ▁se
          - zio
          - str
          - ▁o
          - ▁ma
          - z
          - ▁le
          - da
          - mento
          - ▁fa
          - chi
          - mente
          - ▁nel
          - nza
          - ▁qua
          - ▁ha
          - ▁imp
          - ggi
          - v
          - ▁tu
          - zia
          - h
          - q
          - à
          - ò
          - è
          - k
          - ì
          - é
          - y
          - w
          - x
          - j
          - í
          - ù
          - á
          - '-'
        encoder:
          _target_: nemo.collections.asr.modules.SqueezeformerEncoder
          adaptive_scale: true
          att_context_size:
          - -1
          - -1
          conv_kernel_size: 31
          conv_norm_type: batch_norm
          d_model: 196
          dropout: 0.1
          dropout_att: 0.1
          dropout_emb: 0.0
          feat_in: ${model.preprocessor.features}
          feat_out: -1
          ff_expansion_factor: 4
          n_heads: 4
          n_layers: 18
          pos_emb_max_len: 5000
          self_attention_model: rel_pos
          subsampling: dw_striding
          subsampling_conv_channels: -1
          subsampling_factor: 4
          time_recovery_idx: null
          time_reduce_idx: 7
          untie_biases: true
          xscaling: true
        log_prediction: true
        optim:
          betas:
          - 0.9
          - 0.98
          lr: 0.002
          name: adamw
          sched:
            decay_rate: 1.0
            hold_ratio: 0.3
            hold_steps: null
            min_lr: 1.0e-06
            name: NoamHoldAnnealing
            warmup_ratio: null
            warmup_steps: 10
          weight_decay: 4e-5
        preprocessor:
          _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
          dither: 1.0e-05
          features: 80
          frame_splicing: 1
          log: true
          n_fft: 512
          normalize: per_feature
          pad_to: 0
          pad_value: 0.0
          sample_rate: ${model.sample_rate}
          window: hann
          window_size: 0.025
          window_stride: 0.01
        sample_rate: 16000
        skip_nan_grad: false
        spec_augment:
          _target_: nemo.collections.asr.modules.SpectrogramAugmentation
          freq_masks: 2
          freq_width: 27
          time_masks: 5
          time_width: 0.05
        test_ds:
          batch_size: 8
          manifest_filepath: manifests/prepared/test_manifest.json
          num_workers: 8
          pin_memory: true
          sample_rate: ${model.sample_rate}
          shuffle: false
          use_start_end_token: false
        tokenizer:
          dir: tokenizer_spe_unigram_v128
          type: bpe
        train_ds:
          batch_size: 32
          bucketing_batch_size: null
          bucketing_strategy: synced_randomized
          is_tarred: false
          manifest_filepath: manifests/prepared/train_manifest.json
          max_duration: 30.0
          min_duration: 0.1
          num_workers: 20
          pin_memory: true
          sample_rate: ${model.sample_rate}
          shuffle: true
          shuffle_n: 2048
          tarred_audio_filepaths: null
          trim_silence: false
          use_start_end_token: false
        validation_ds:
          batch_size: 8
          manifest_filepath: manifests/prepared/dev_manifest.json
          num_workers: 8
          pin_memory: true
          sample_rate: ${model.sample_rate}
          shuffle: false
          use_start_end_token: false
      name: Squeezeformer_ctc_bpe_EUP_Small_tokenizer-spe-unigram-128
      target: nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE
      trainer:
        accelerator: auto
        accumulate_grad_batches: 1
        benchmark: false
        check_val_every_n_epoch: 1
        devices: -1
        enable_checkpointing: false
        enable_progress_bar: true
        gradient_clip_val: 0.0
        log_every_n_steps: 10
        logger: false
        max_epochs: 1000
        max_steps: -1
        num_nodes: 1
        num_sanity_val_steps: 0
        precision: 32
        resume_from_checkpoint: null
        strategy: ddp
        sync_batchnorm: true
        val_check_interval: 1.0


I'm performing the training with the [/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py](https://github.com/NVIDIA/NeMo/blob/2ce45369f7ab6cd20c376d1ed393160f5e54be0c/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py) script.

Any suggestion?

Thanks.