language model training for asr
State: closed | Pull request: no
Labels: name: stale

hello.
while trying to train a language model for asr with train_kenlm.py, i encounter the following error:
''
[NeMo W 2023-05-06 17:30:53 optimizers:54] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-05-06 17:30:53 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-05-06 17:30:54 nemo_logging:349] /root/anaconda3/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
    See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
      ret = run_job(

[NeMo I 2023-05-06 17:30:54 kenlm_utils:73] Loading nemo model '/home/azad2/NeMo/Model-fa-512.nemo' ...
[NeMo I 2023-05-06 17:30:54 mixins:170] Tokenizer SentencePieceTokenizer initialized with 700 tokens
[NeMo W 2023-05-06 17:30:54 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config :
    manifest_filepath: /home/azad/manifests/commonvoice_train_manifest.json,/home/azad/manifests/commonvoice_dev_manifest.json
    sample_rate: 16000
    batch_size: 32
    trim_silence: true
    max_duration: 16.7
    shuffle: true
    is_tarred: false
    tarred_audio_filepaths: null
    num_workers: 8
    pin_memory: true
    use_start_end_token: true

[NeMo W 2023-05-06 17:30:54 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
    Validation config :
    manifest_filepath: /home/azad/manifests/commonvoice_test_manifest.json
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: true

[NeMo W 2023-05-06 17:30:54 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config :
    manifest_filepath: /home/azad/manifests/commonvoice_test_manifest.json
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: true

[NeMo I 2023-05-06 17:30:54 features:291] PADDING: 16
Call to cuInit results in CUDA_ERROR_NO_DEVICE
[NeMo I 2023-05-06 17:30:55 save_restore_connector:249] Model EncDecCTCModelBPE was successfully restored from /home/azad2/NeMo/Model-fa-512.nemo.
[NeMo I 2023-05-06 17:30:55 train_kenlm:149] Running lmplz command

    /home/azad2/NeMo/decoders/kenlm/build/bin/lmplz -o 6 --arpa /NeMo/model.tmp.arpa --discount_fallback --prune 0


/home/azad2/NeMo/decoders/kenlm/util/file.cc:86 in int util::CreateOrThrow(const char*) threw ErrnoException because `-1 == (ret = open(name, 0100 | 01000 | 02, 0400 | 0200 | (0400 >> 3) | ((0400 >> 3) >> 3)))'.
No such file or directory while creating /NeMo/model.tmp.arpa
Error executing job with overrides: ['nemo_model_file=/home/azad2/NeMo/Model-fa-512.nemo', 'train_paths=[/home/azad2/NeMo/commonvoice_train_manifest_processed.JSON]', 'kenlm_bin_path=/home/azad2/NeMo/decoders/kenlm/build/bin', 'kenlm_model_file=/NeMo/model', 'ngram_length=6']
Traceback (most recent call last):
  File "/home/azad2/NeMo/scripts/asr_language_modeling/ngram_lm/train_kenlm.py", line 164, in main
    raise RuntimeError("Training KenLM was not successful!")
RuntimeError: Training KenLM was not successful!

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
''
i dont know whats the problem.
any idea whats the problem?
thanks in advance.