Canary model stuck in a loop? Just repeats the same phrases over and over.
State: closed | Pull request: no
Labels: name: bug

name: stale

I recorded a 5 minute audio segment on my iPhone 11, at 24 bit / 48Khz, using this as my json input:

{ "audio_filepath": "/ldir/test.wav", "duration": "None", "taskname": "asr", "source_lang": "en", "target_lang": "en", "pnc": "yes", "answer": "na" }

It was just a test recording of me reading from Vol 1 of The Art of Computer Programming. The return from the call to
```
canary_model.transcribe("input_manifest.json", batch_size=16)
```
looks like
```
['Historically, the first interpreters were built around machine like languages designed in a single word. In such a case, there is another way to pick out the appropriate interpretive language, which is designed in a single word. Such a programme may be considered as a longer interpretive language. In such a language, there is a language that is designed in a single word. Interpretive language. In such a language, there is a language designed in a single word design. Interpretive language designed in a single word design. In such a word design, there is a language designed interpreter designed in a single word design. In such a word designed interpreter design, designed in a single word design. In such a word designed interpreter design, designed in a single word designation design interpreter designation designation designed in a single word design designation designed interprect design design designation designed inter designation designation design designed inter designation designation designed inter designation designation designation designed inter interprect designation designation designed inter interprect designation designation designed in designed interprect designed interprection designed interprection designed interprect designed interprect designed interprection designed interprection designed inter designed inter']
```

You can see it starts off OK, but this it just gets stuck in a loop repeating the same word or phrase, and it's much shorter than the audio segment. I have seen this repeatedly in different tests, and I have no idea what's causing it.

I'm using the following to launch the container
```
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -it --rm -v /mnt/c/Users/bcollins/local_directory:/ldir nvcr.io/nvidia/nemo:24.01.speech
```

on an RTX 4090 mobile chipset in a Windows laptop. VRAM is 16GB.

I am facing a similar issue. Any help would be appreciated. Thank you.

I am afraid to say this is common with AED models, and this scenario is called hallucination. 

Would it be possible for you to share the audio? @pzelasko FYI

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Sorry for a late answer. Please try to process that file with this script instead: https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py

it will slice your long segment to shorter chunks and run inference on them. The result should be much better. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.