[Question] Low accuracy in VAD demo?
State: closed | Pull request: no

I ran the [VAD demo](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb) and noticed that the prediction accuracy on the [example audio](https://dldata-public.s3.us-east-2.amazonaws.com/VAD_demo.wav) seems quite low (see chart below).
Do you see a similar output when you run the notebook?

I changed the script in one place - I commented out these lines from function `offline_inference`, because my host doesn't have an audio device:
```
    stream = p.open(format=p.get_format_from_width(wf.getsampwidth()),
                    channels=CHANNELS,
                    rate=RATE,
                    output = True)
```

![image](https://user-images.githubusercontent.com/39620641/104517221-ef980d00-55f5-11eb-82ff-dc26c5cfe4f3.png)



**Environment overview**

 - Environment location: AWS EC2
 - Method of NeMo install: `!python3 -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]`

**Environment details**

- OS version Ubuntu 20.04.1 LTS (Focal Fossa)
- PyTorch version 1.6.0
- Python version 3.8.5

**Additional context**

GPU model: Tesla T4

Hi @skubicius Thanks for your question 😃 . This is the expected plot for this demo wav. There are a few reasons for a not ideal plot for the pre-trained models, such as data and sample rate mismatch, diverse recording (microphone) environment, language, etc. And please note that metrics would be different for different use cases, for example, should we treat short pause as no-speech? It depends. We would encourage you to further tune the model (such as model finetuning, threshold optimization) on your training/development data to boost the performance on unseen data. In the meanwhile, we are trying to release better pre-trained models and will update the notebook with the necessary clarification and guide. 

 Hi @fayejf, thank you for the explanation, I'll check out model tuning.

No prob! @skubicius  You should be able to find instructions in [tutorial 06 ](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb). Please let us know if you have more questions.

hi @fayejf 
I trained VAD model using my own dataset follow instructions in [tutorial 6](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb)
My dataset is Vietnamese telephony conversation, 15 hours with 8 hours of speech and 7 hours of noise. I got accuracy 95% on my test set.
The format of dataset quite similar with google speech dataset. Each sample of speech is one statement and have duration 2s to 5s.

When i follow instruction in [tutorial 7](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb) to inference sample in test set what contain only one statement, the prediction is good. The problem is when i inference long audio has duration 20s of speech and noise, the prediction is really bad.
This is predict sample of speech has only one statement 

![foo](https://user-images.githubusercontent.com/83386486/121887072-78031100-cd40-11eb-8858-302703bbf751.png)

And long audio, 0 to 5s of audio is noise.

![bar](https://user-images.githubusercontent.com/83386486/121887987-a59c8a00-cd41-11eb-8c5d-14383dbcb517.png)

1. Any idea to deal with this problem
2. I want to tune threshold but it required RTTM ground truth, what is the format of RTTM file and how to generate this file from dataset like google speech.


@guyqaz 

Thanks for your question. 
What is the sample rate of your long audio sample? Looks like it misses some predictions after 15s.

You could find the definition of NIST Rich Transcription Time Marked (RTTM) format [here](https://web.archive.org/web/20170119114252/http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf). We have prepared an example file for you [here](https://nemo-public.s3.us-east-2.amazonaws.com/an4_diarize_test.rttm).  For google speech basically you can have something like.

`SPEAKER File1 1 0.5 0.671 <NA> <NA> Speaker1 <NA> <NA>`
`SPEAKER File1 1 1.7 1.258 <NA> <NA> Speaker1 <NA> <NA>`

To tune threshold, the RTTM file **is not required**. If you have frame level label, for example, [1,1,1,1,1,0,0,0,0,1,1,1], you can directly use that as well.  

Note the flow is 
frame level prediction (w/ or wo/posterior)  -> to convert to speech / no-speech given **threshold**  -> compare with ground truth label to find optimal threshold


Hi @fayejf , The sample rate of long audio sample is 16000Hz.
Now i am defining one audio of sentence as speech, not a word level like google speech dataset. But i still using [process_vad_data](https://github.com/NVIDIA/NeMo/blob/main/scripts/dataset_processing/process_vad_data.py) to generate manifest file.
When infer audio just has statement or noise, i think it good because almost all frame are predicted to be speech or all noise. But when predict long audio have both speech and noise. Model dose not seen recognize speech and noise.

My dataset also have label with shorter duration what label for stop point, noise in audio contain speech of statement. 
- Do you think i should using shorter label or just using audio of sentence is speech.
- Is there a tutorial to tune threshold from output of predict frame level [vad_infer](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/vad_infer.py) and manifest file.

@guyqaz 
Answered here https://github.com/NVIDIA/NeMo/issues/2353#issuecomment-861862612