Forced Aligner with long audios
State: closed | Pull request: no

Hello! when trying to do a forced alignment with long audios (1h 30m+) I get the following exception:

`torch.cuda.OutOfMemoryError: CUDA out of memory. ...`

Now I understand it might have something to do with batch size or chunk size, or even it can just be an issue of not having enough GPU memory (I'm running it on google colab, which has 15GB)

What should be the correct approach to handle large audios?

for the record, this is how I'm running the align script:

```
!HYDRA_FULL_ERROR=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python $NEMO_DIR_PATH/tools/nemo_forced_aligner/align.py \
  pretrained_name="stt_en_fastconformer_hybrid_large_pc" \
  manifest_filepath=$manifest_filepath \
  output_dir=$WORK_DIR/nfa_output/ \
  total_buffer_in_secs=2 \
  chunk_batch_size=16 \
  chunk_len_in_secs=0.8
``

use ctc models for aligning very long audio duration files. 

You can use hybrid models in CTC mode automatically. @erastorgueva-nv could you advise 

It should work if you also add the parameter `use_local_attention=true` (reference: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tools/nemo_forced_aligner.html#optional-parameters)

I tried using the model `stt_en_conformer_ctc_small` and got the same result, then I tried playing with the Viterbi device and the transcriber device values, passing both of those values as CPU killed the process, no error log or anything, just this:
`Transcribing:   0% 0/1 [00:00<?, ?it/s]^C`

When passing `transcribe_device='cuda'` I got the CUDA OOM error again.
When passing `viterbi_device='cpu'` I got the following output again: `Transcribing:   0% 0/1 [00:00<?, ?it/s]^C`

Then I tried running the command using the following model:

`stt_en_conformer_ctc_small`

here I got a different error shared below:

```
!HYDRA_FULL_ERROR=1 python $NEMO_DIR_PATH/tools/nemo_forced_aligner/align.py \
  pretrained_name="stt_en_conformer_ctc_small" \
  manifest_filepath=$manifest_filepath \
  output_dir=$WORK_DIR/nfa_output/ \
  use_local_attention=true \
  transcribe_device='cuda' \
  viterbi_device='cuda'
```

Logs:

```
[NeMo I 2024-04-11 23:21:25 features:289] PADDING: 0
[NeMo I 2024-04-11 23:21:26 save_restore_connector:263] Model EncDecCTCModelBPE was successfully restored from /root/.cache/torch/NeMo/NeMo_1.23.0rc0/stt_en_conformer_ctc_small/5d2d8e5b2b5adb8f5091363c6ba19c55/stt_en_conformer_ctc_small.nemo.
[NeMo I 2024-04-11 23:21:26 align:250] Flag use_local_attention is set to True => will try to use local attention for model if it allows it
Transcribing: 100% 1/1 [00:06<00:00,  6.46s/it]
Error executing job with overrides: ['pretrained_name=stt_en_conformer_ctc_small', 'manifest_filepath=WORK_DIR/manifest.json', 'output_dir=WORK_DIR/nfa_output/', 'use_local_attention=true', 'transcribe_device=cuda', 'viterbi_device=cuda']
Traceback (most recent call last):
  File "/content/NeMo/tools/nemo_forced_aligner/align.py", line 352, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/config/hydra_runner.py", line 129, in wrapper
    _run_hydra(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/content/NeMo/tools/nemo_forced_aligner/align.py", line 318, in main
    (log_probs_batch, y_batch, T_batch, U_batch, utt_obj_batch, output_timestep_duration,) = get_batch_variables(
  File "/content/NeMo/tools/nemo_forced_aligner/utils/data_prep.py", line 748, in get_batch_variables
    utt_obj = get_utt_obj(
  File "/content/NeMo/tools/nemo_forced_aligner/utils/data_prep.py", line 320, in get_utt_obj
    word_tokens_cased = restore_token_case(word, word_tokens)
  File "/content/NeMo/tools/nemo_forced_aligner/utils/data_prep.py", line 164, in restore_token_case
    if token_char == word[word_char_pointer]:
IndexError: string index out of range
```


Is there a way to know how many resources we need to run any model?

After some trial and error, I figured out my text had some characters that didn't belong to the transcription.

I was able to align the 1h 30m audio I needed.

Thanks for the help!

Glad you were able to get it to work! A pointer about the command you posted:
```
!HYDRA_FULL_ERROR=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python $NEMO_DIR_PATH/tools/nemo_forced_aligner/align.py \
  pretrained_name="stt_en_fastconformer_hybrid_large_pc" \
  manifest_filepath=$manifest_filepath \
  output_dir=$WORK_DIR/nfa_output/ \
  total_buffer_in_secs=2 \
  chunk_batch_size=16 \
  chunk_len_in_secs=0.8
```
^ in the above command, buffered chunk streaming is not being done (and the params `total_buffer_in_secs`, `chunk_batch_size`, `chunk_len_in_secs` are being ignored). If you want to use it, you need to add `use_buffered_chunked_streaming=true`. But seems like you do not need it.