VAD inference update
State: closed | Pull request: yes

Update VAD inference to make it easy to combine with Speaker Diarization and more user friendly (user would like txt output) 

Add/update:
1. Truncate long audio files to separate json line to avoid CUDA memory issue
2. Frame level inference similar to speech_to_text_infer.py because the frame level streaming is kinda different with asr and speech command.
3. A posterior including overlapped prediction and smoothing filter
4. Write to RTTM like file.

AUDIO -> MANIFEST -> VAD OUTPUT TXT -> VAD RTTM-like TABLE

Signed-off-by: fayejf <fayejf07@gmail.com>

This pull request **introduces 5 alerts** when merging c0ea1dedb51f1613303a24005612c44e29f0b303 into aa4541b91341e0dda5300f1ded5f60cbbc74429a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6f4f2f933f3cfd83e9c80db0d46a48fe806dd015)

**new alerts:**

* 3 for Unused import
* 1 for Variable defined multiple times
* 1 for Nested loops with same variable

This pull request **introduces 5 alerts** when merging d95efb621ad7ec3fff48b316117116e109cd32db into aa4541b91341e0dda5300f1ded5f60cbbc74429a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-893456f8b20f71b9e8ff2b4f64b83827f7cc5b98)

**new alerts:**

* 3 for Unused import
* 1 for Variable defined multiple times
* 1 for Nested loops with same variable

This pull request **introduces 5 alerts** when merging 4711223a9807aadbd261fd1b7557d70cf3c0ad42 into 75774cf24755bddb393e3b1a4cbcc1921daf8898 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ad317e1a2ba8c643811447b8f69be1e135afd3d7)

**new alerts:**

* 3 for Unused import
* 1 for Variable defined multiple times
* 1 for Nested loops with same variable

please don't final review/merge it.
Has one last thing to update.

Take care of JOINT of long audio files

This pull request **introduces 3 alerts** when merging fcf03b25f2241fc0792b7ffad9426f74a488bf79 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-710fb58d744a94eec3e72c926bd7982838af9294)

**new alerts:**

* 2 for Unused local variable
* 1 for Nested loops with same variable

Hey @fayejf, appreciate all the work and progress!! Nemo seems to be, at least to me, one of the most accessible and complete end-to-end ASR systems out there, wrt ASR and downstream NLP tasks. 

So I've been deep diving into the ASR world and NeMo, and came across this PR when checking to see if Nemo supports diarization. According to https://github.com/NVIDIA/NeMo/issues/1299 , it seems like you don't currently? But from here https://github.com/NVIDIA/NeMo/commit/c7656318c260d5c33422e93af248e314673b36cf - it seems like it might be something you guys are working on already? Would I be correct in thinking `spkr_get_emb.py` has pretty much most of what I need to be able to perform diarization? 

> Hey @fayejf, appreciate all the work and progress!! Nemo seems to be, at least to me, one of the most accessible and complete end-to-end ASR systems out there, wrt ASR and downstream NLP tasks.
> 
> So I've been deep diving into the ASR world and NeMo, and came across this PR when checking to see if Nemo supports diarization. According to #1299 , it seems like you don't currently? But from here [c765631](https://github.com/NVIDIA/NeMo/commit/c7656318c260d5c33422e93af248e314673b36cf) - it seems like it might be something you guys are working on already? Would I be correct in thinking `spkr_get_emb.py` has pretty much most of what I need to be able to perform diarization?

Hi @ShantanuNair Thanks for your interest in NeMo. We are working on diarization 😄  and currently we have voice activity detection, speaker recognition and speaker verification.  For diarization you will also need clustering. 
Speech detection ✅ -> speaker embedding  extraction ✅ (yes spkr_get_emb.py) -> clustering 🚗 -> (optional) resegmentation .  
 
Hope this helps.

This pull request **introduces 4 alerts** when merging c95ce6b2cd0d98e727550c21e38007101352e9ea into 38c854da3f553b6577d6f8b08490e026b53b7ea3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e094b7b495eb1b4f4a5349d14fabe850ed5f1da2)

**new alerts:**

* 2 for Unused local variable
* 1 for Redundant assignment
* 1 for Nested loops with same variable

@jbalam-nv @nithinraok  Please review the changes. Thanks! 🥰 

FYI @ShantanuNair The PR is merged. Feel free to play with the scripts. Questions and feedbacks are very welcome!

> FYI @ShantanuNair The PR is merged. Feel free to play with the scripts. Questions and feedbacks are very welcome!

Thank you! Appreciate the openness. Will reach out if needed :)

Hey @fayejf i am confused with Speaker  diarization. How to perform a simple inference of Speaker  diarization on a given wav file and get outputs in formats like:
```
speaker 1    0.0 to 0.15
speaker 2    0.15 to 0.30
speaker 4    0.30 to 0.45
speaker 1    0.45 to 1.00
speaker 3    1.00 to 1.15

```
where values on right indicates time of uterance
I am aware of `07_Online_Offline_Microphone_VAD_Demo.ipynb` but there i am getting the result of just the vad. But not in the form of `who spoke when` as mentioned above

Hi @jainal09 Thanks for your interest in NeMo.
We don't have an inference of speaker diarization and we are working on it and will release it soon. 😄  
Currently, we have voice activity detection, speaker recognition, and speaker verification. For diarization, you will also need clustering.

Speech detection ✅ (This PR)-> speaker embedding extraction ✅ (spkr_get_emb.py ,etc.) -> clustering 🚗  (WIP) -> (optional) resegmentation

Hey @fayejf thanks for replying.
I really appreciate all the work and progress that you guys are putting in nemo.
I will keep around and eagerly waiting for inference of speaker diarization to get released.
Really excited for this!😊

 🥰  Thanks!  @jainal09 ❤️

Hey @fayejf @nithinraok  i also have one more question in the events of overlapped speech will this work to recognize both speakers or will it flag it as new speaker?

Thanks for your question! @jainal09  It's currently a work-in-progress. Please wait for the release. Feel free to leave questions but we might not have clear answers now.

Sure! Thanks @fayejf I will wait till release 🤞.

Hi @fayejf any updates on this?
Also, I had the same questions that @serg06 mentioned here : https://github.com/NVIDIA/NeMo/issues/1299#issuecomment-739462587
As to performing speaker diarialization as per the method mentioned by you above:
> Speech detection ✅ (This PR)-> speaker embedding extraction ✅ (spkr_get_emb.py ,etc.) -> clustering 🚗 (WIP) -> (optional) resegmentation


Hey @fayejf pls reply any updates diarialization on the  script?

@jainal09 Wait. It's holiday season as well, so expect things to slow down.