My VAD performance is terrible!
State: closed | Pull request: no

### Discussed in https://github.com/NVIDIA/NeMo/discussions/6700

<div type='discussions-op-text'>

<sup>Originally posted by **great-goblin** May 22, 2023</sup>
I'm trying to get a VAD model to work for diarization inference, but the VAD model alone has atrocious performance. I tried following along using the parameters [given in this tutorial](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb#scrollTo=4r4XOzhWgj81), but it had very high error rates (95% diarization ER, 80% miss). I've had much better success with using a [basic ELAN silence recognizer script,](https://www.mpi.nl/corpus/html/elan/ch02s04s03.html) but I'm not experienced enough with the NeMo toolkit to know how to substitute in that 'correct' VAD segmentation. I'm just trying to get the basic System VAD (marblenet) -> segmentation -> embedding extractor -> clustering -> neural diarizer (MSDD) to work.

[This links to a graphic](https://www.imagehost.at/image/2JI7b) I made to show how VAD is currently doing with my .wav fle (red line is ideal segmentation).

The .wav file I'm looking at has been resampled to 16kHz, and features a pilot and air traffic controller speaking to each other. To give a sense of the back-and-forth, this is the .rttm:

```SPEAKER WTAN4 1 0.920 2.75 <NA> <NA> B <NA>
SPEAKER WTAN4 1 4.020 3.1800000000000006 <NA> <NA> A <NA>
SPEAKER WTAN4 1 7.700 1.4999999999999991 <NA> <NA> B <NA>
SPEAKER WTAN4 1 11.340 2.2799999999999994 <NA> <NA> B <NA>
SPEAKER WTAN4 1 16.080 6.780000000000001 <NA> <NA> A <NA>
SPEAKER WTAN4 1 25.340 0.8900000000000006 <NA> <NA> B <NA>
SPEAKER WTAN4 1 26.400 4.16 <NA> <NA> A <NA>
SPEAKER WTAN4 1 31.000 4.185000000000002 <NA> <NA> B <NA>
SPEAKER WTAN4 1 35.380 4.559999999999995 <NA> <NA> A <NA>
SPEAKER WTAN4 1 40.480 3.1200000000000045 <NA> <NA> B <NA>
```

1. I start by initializing and importing everything per the tutorial linked above. My .rttm's look fine in Annotation format, no issues or any other issues (warnings aside).

2. I load the diar_infer_telephonic yaml into Omega and modify the following settings. Mostly I followed the tutorial, except I've made more modifications to the diarizer parameters in an attempt to make the VAD look better. 
```python
MODEL_CONFIG = os.path.join(data_dir,'diar_infer_telephonic.yaml')
config = OmegaConf.load(MODEL_CONFIG)
config.diarizer.manifest_filepath = 'data/input_manifest.json'
config.diarizer.out_dir = output_dir # Directory to store intermediate files and prediction outputs
pretrained_speaker_model = 'titanet_large'
config.diarizer.speaker_embeddings.model_path = pretrained_speaker_model
config.diarizer.speaker_embeddings.parameters.window_length_in_sec = [1.5,1.25,1.0,0.75,0.5] 
config.diarizer.speaker_embeddings.parameters.shift_length_in_sec = [0.75,0.625,0.5,0.375,0.1] 
config.diarizer.speaker_embeddings.parameters.multiscale_weights= [1,1,1,1,1] 
config.diarizer.oracle_vad = True # ----> ORACLE VAD 
config.diarizer.clustering.parameters.oracle_num_speakers = False
config.diarizer.msdd_model.model_path = 'diar_msdd_telephonic' # Telephonic speaker diarization model 
config.diarizer.msdd_model.parameters.sigmoid_threshold = [0.7, 1.0] # Evaluate with T=0.7 and T=1.0


pretrained_vad = 'vad_multilingual_marblenet'

config.num_workers = 1 # Workaround for multiprocessing hanging with ipython issue 
output_dir = os.path.join(ROOT, 'outputs')
config.diarizer.manifest_filepath = 'data/input_manifest.json'
config.diarizer.out_dir = output_dir #Directory to store intermediate files and prediction outputs
config.diarizer.speaker_embeddings.model_path = pretrained_speaker_model
config.diarizer.oracle_vad = False # compute VAD provided with model_path to vad config
config.diarizer.clustering.parameters.oracle_num_speakers=False

#VAD parameters - this is what I've edited
config.diarizer.vad.model_path = pretrained_vad
config.diarizer.vad.parameters.window_length_in_sec = 0.5
config.diarizer.vad.parameters.shift_length_in_sec = 0.03
config.diarizer.vad.parameters.smoothing = 'median'
config.diarizer.vad.parameters.overlap = 0.9
config.diarizer.vad.parameters.onset = 0.05
config.diarizer.vad.parameters.offset = 0.7
config.diarizer.vad.parameters.pad_onset = 0.05
config.diarizer.vad.parameters.pad_offset = -0.09
config.diarizer.vad.parameters.min_duration_on = 0.15
config.diarizer.vad.parameters.min_duration_off = 0.05
```

3. Then, I run the diarizer and visualize the results.

```python
from nemo.collections.asr.models import ClusteringDiarizer
sd_model = ClusteringDiarizer(cfg=config)
sd_model.diarize()
from nemo.collections.asr.parts.utils.vad_utils import plot

if config.diarizer.vad.parameters.smoothing:
    vad_output_filepath = f'{output_dir}/vad_outputs/overlap_smoothing_output_median_{config.diarizer.vad.parameters.overlap}/{filename}.{config.diarizer.vad.parameters.smoothing}'
else:
    vad_output_filepath = f'{output_dir}/vad_outputs/{filename}.frame'

plot(
    an4_audio,
    vad_output_filepath, 
    None,
    per_args = config.diarizer.vad.parameters, #threshold
    ) 
# Save the plot as an image
plt.savefig(f'vad_plot.png')
```
I've played a lot with the VAD parameters in step 2 above, but I'm finding it frustrating that a simple ELAN script from two decades ago does a better job at VAD than this sophisticated model. I'm confident it's my own inexperience with this model, so please tell me how I can improve! Ultimately I'd like to actually diarize this model, and if I can figure out how, somehow fine-tune it on my own data, but any suggestions at all would be welcome!

</div>