lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*) threw SpecialWordMissingException.
State: closed | Pull request: no
Labels: name: stale

hello
while trying to train the language model with train-kenlm.py for asr, i encounter the following error:
"
[NeMo W 2023-05-08 14:55:20 optimizers:54] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-05-08 14:55:21 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-05-08 14:55:21 nemo_logging:349] /root/anaconda3/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
    See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
      ret = run_job(

[NeMo I 2023-05-08 14:55:21 kenlm_utils:73] Loading nemo model '/home/azad2/NeMo/Model-fa-512.nemo' ...
[NeMo I 2023-05-08 14:55:21 mixins:170] Tokenizer SentencePieceTokenizer initialized with 700 tokens
[NeMo W 2023-05-08 14:55:21 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config :
    manifest_filepath: /home/azad/manifests/commonvoice_train_manifest.json,/home/azad/manifests/commonvoice_dev_manifest.json
    sample_rate: 16000
    batch_size: 32
    trim_silence: true
    max_duration: 16.7
    shuffle: true
    is_tarred: false
    tarred_audio_filepaths: null
    num_workers: 8
    pin_memory: true
    use_start_end_token: true

[NeMo W 2023-05-08 14:55:21 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
    Validation config :
    manifest_filepath: /home/azad/manifests/commonvoice_test_manifest.json
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: true

[NeMo W 2023-05-08 14:55:21 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config :
    manifest_filepath: /home/azad/manifests/commonvoice_test_manifest.json
    sample_rate: 16000
    batch_size: 8
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: true
    trim_silence: true

[NeMo I 2023-05-08 14:55:21 features:291] PADDING: 16
Call to cuInit results in CUDA_ERROR_NO_DEVICE
[NeMo I 2023-05-08 14:55:22 save_restore_connector:249] Model EncDecCTCModelBPE was successfully restored from /home/azad2/NeMo/Model-fa-512.nemo.
[NeMo I 2023-05-08 14:55:22 train_kenlm:150] Running lmplz command

    /home/azad2/NeMo/decoders/kenlm/build/bin/lmplz -o 6 --arpa /home/azad2/NeMo/model.tmp.arpa --discount_fallback --prune 0


=== 1/5 Counting and sorting n-grams ===
File stdin isn't normal.  Using slower read() instead of mmap().  No progress bar.
Unigram tokens 0 types 3
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:36 2:3314737408 3:6215132672 4:9944212480 5:14501976064 6:19888424960
Substituting fallback discounts for order 0: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 1: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 2: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 3: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 4: D1=0.5 D2=1 D3+=1.5
Substituting fallback discounts for order 5: D1=0.5 D2=1 D3+=1.5
Statistics:
1 2 D1=0.5 D2=1 D3+=1.5
2 0 D1=0.5 D2=1 D3+=1.5
3 0 D1=0.5 D2=1 D3+=1.5
4 0 D1=0.5 D2=1 D3+=1.5
5 0 D1=0.5 D2=1 D3+=1.5
6 0 D1=0.5 D2=1 D3+=1.5
Memory estimate for binary LM:
type       B
probing  144 assuming -p 1.5
probing  172 assuming -r models -p 1.5
trie     169 without quantization
trie    9366 assuming -q 8 -b 8 quantization
trie     261 assuming -a 22 array pointer compression
trie    9458 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
=== 3/5 Calculating and sorting initial probabilities ===
Chain sizes: 1:24 2:16 3:20 4:24 5:28 6:32
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:24 2:16 3:20 4:24 5:28 6:32
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

=== 5/5 Writing ARPA model ===
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

Name:lmplz      VmPeak:52790308 kB      VmRSS:5660 kB   RSSMax:8306852 kB       user:0.198719   sys:1.42085     CPU:1.61958     real:1.63664
[NeMo I 2023-05-08 14:55:24 train_kenlm:175] Running binary_build command

    /home/azad2/NeMo/decoders/kenlm/build/bin/build_binary trie /home/azad2/NeMo/model.tmp.arpa /home/azad2/NeMo/model


Reading /home/azad2/NeMo/model.tmp.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
/home/azad2/NeMo/decoders/kenlm/lm/vocab.cc:305 in void lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*) threw SpecialWordMissingException.
The ARPA file is missing 
 and the model is configured to reject these models.  Run build_binary -s to disable this check. Byte: 96
ERROR
Error executing job with overrides: ['nemo_model_file=/home/azad2/NeMo/Model-fa-512.nemo', 'train_paths=[/home/azad2/NeMo/commonvoice_train_manifest_processed.JSON]', 'kenlm_bin_path=/home/azad2/NeMo/decoders/kenlm/build/bin', 'kenlm_model_file=/home/azad2/NeMo/model', 'ngram_length=6', 'preserve_arpa=true']
Traceback (most recent call last):
  File "/home/azad2/NeMo/scripts/asr_language_modeling/ngram_lm/train_kenlm.py", line 179, in main
    raise RuntimeError("Training KenLM was not successful!")
RuntimeError: Training KenLM was not successful!
"
i really need your help.thanks!