ASR TypeError: '<' not supported between instances of 'NoneType' and 'int'
State: closed | Pull request: no
Labels: name: bug



!bash start_train_ctc.sh 
[NeMo W 2024-04-05 16:16:10 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
    See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
      ret = run_job(
    
[NeMo I 2024-04-05 16:16:10 speech_to_text_ctc_bpe:99] Hydra config: name: Conformer-CTC-BPE-Large
    model:
      sample_rate: 16000
      log_prediction: true
      ctc_reduction: mean_batch
      skip_nan_grad: false
      train_ds:
        manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/train.json
        sample_rate: ${model.sample_rate}
        batch_size: 24
        shuffle: true
        num_workers: 16
        pin_memory: true
        use_start_end_token: false
        trim_silence: false
        max_duration: 30
        min_duration: 0.1
        is_tarred: false
        tarred_audio_filepaths: null
        shuffle_n: 2048
        bucketing_strategy: synced_randomized
        bucketing_batch_size: 8
      validation_ds:
        manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/validated.json
        sample_rate: ${model.sample_rate}
        batch_size: 1
        max_duration: 30
        shuffle: false
        num_workers: 8
        pin_memory: true
        use_start_end_token: false
      test_ds:
        manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/test.json
        sample_rate: ${model.sample_rate}
        batch_size: 4
        shuffle: false
        num_workers: 8
        pin_memory: true
        use_start_end_token: false
      tokenizer:
        dir: /content/drive/MyDrive/1T
        type: bpe
      preprocessor:
        _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
        sample_rate: ${model.sample_rate}
        normalize: per_feature
        window_size: 0.025
        window_stride: 0.01
        window: hann
        features: 80
        n_fft: 512
        log: true
        frame_splicing: 1
        dither: 1.0e-05
        pad_to: 0
        pad_value: 0.0
      spec_augment:
        _target_: nemo.collections.asr.modules.SpectrogramAugmentation
        freq_masks: 2
        time_masks: 5
        freq_width: 27
        time_width: 0.05
      encoder:
        _target_: nemo.collections.asr.modules.ConformerEncoder
        feat_in: ${model.preprocessor.features}
        feat_out: -1
        n_layers: 17
        d_model: 512
        subsampling: striding
        subsampling_factor: 4
        subsampling_conv_channels: -1
        ff_expansion_factor: 4
        self_attention_model: rel_pos
        n_heads: 8
        att_context_size:
        - -1
        - -1
        xscaling: true
        untie_biases: true
        pos_emb_max_len: 5000
        conv_kernel_size: 31
        conv_norm_type: batch_norm
        dropout: 0.1
        dropout_emb: 0.0
        dropout_att: 0.1
      decoder:
        _target_: nemo.collections.asr.modules.ConvASRDecoder
        feat_in: null
        num_classes: -1
        vocabulary: []
      optim:
        name: adamw
        lr: 0.5
        betas:
        - 0.9
        - 0.98
        weight_decay: 0.001
        sched:
          name: NoamAnnealing
          d_model: ${model.encoder.d_model}
          warmup_steps: 10000
          warmup_ratio: null
          min_lr: 1.0e-06
    trainer:
      devices: -1
      num_nodes: 1
      max_epochs: 30
      max_steps: null
      val_check_interval: 1.0
      accelerator: gpu
      strategy: ddp
      accumulate_grad_batches: 1
      gradient_clip_val: 0.0
      precision: 32
      log_every_n_steps: 100
      num_sanity_val_steps: 0
      check_val_every_n_epoch: 1
      sync_batchnorm: true
      enable_checkpointing: false
      logger: false
      benchmark: false
    exp_manager:
      exp_dir: null
      name: ${name}
      create_tensorboard_logger: true
      create_checkpoint_callback: true
      checkpoint_callback_params:
        monitor: val_wer
        mode: min
        save_top_k: 10
        always_save_nemo: true
      resume_if_exists: false
      resume_ignore_no_checkpoint: false
      create_wandb_logger: true
      wandb_logger_kwargs:
        name: conformer_large_filtered_english
        project: nemo_conformer
    init_from_pretrained_model: /root/ekstep/nemo_exp/vakyansh-nemo-experimentation/scripts/finetuning/nemo_experiments/Conformer-CTC-BPE-Large/2022-06-01_19-16-48/checkpoints/Conformer-CTC-BPE-Large.nemo
    
Error executing job with overrides: ['model.train_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/train.json', 'model.validation_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/validated.json', 'model.test_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/test.json', 'model.tokenizer.dir=/content/drive/MyDrive/1T', 'model.tokenizer.type=bpe', 'trainer.devices=-1', 'trainer.accelerator=gpu', 'trainer.strategy=ddp', 'exp_manager.create_wandb_logger=True', 'exp_manager.wandb_logger_kwargs.name=conformer_large_filtered_english', 'exp_manager.wandb_logger_kwargs.project=nemo_conformer', '+init_from_pretrained_model=/root/ekstep/nemo_exp/vakyansh-nemo-experimentation/scripts/finetuning/nemo_experiments/Conformer-CTC-BPE-Large/2022-06-01_19-16-48/checkpoints/Conformer-CTC-BPE-Large.nemo']
Traceback (most recent call last):
  File "/content/vakyansh-nemo-experimentation/scripts/finetuning/../../src/finetuning/speech_to_text_ctc_bpe.py", line 101, in main
    trainer = pl.Trainer(**cfg.trainer)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 418, in __init__
    self.fit_loop.epoch_loop = _TrainingEpochLoop(self, min_steps=min_steps, max_steps=max_steps)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py", line 59, in __init__
    if max_steps < -1:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

please somebody help to get this error @titu1994 sir help

In your config use trainer.max_steps = -1 rather than None

thanks sir 🙌