Speaker diarization in NeMO?
State: closed | Pull request: no
Labels: name: feature

**Describe your question**

I would like to write and train a model in Nemo which, given a ground truth labelling of where I think speech or silence is, returns a list of start and end indices in the audio array for each period of detected speech.  

Is this possible?

Also it would be great if I could detect how many people speak on the recording, and label each speech segment with the ID (1,2,3) of the speaker.

Something similar to this: https://github.com/pyannote/pyannote-audio

We don't have (yet) speaker diarization models yet.

This task: "returns a list of start and end indices in the audio array for each period of detected speech" sounds like voice activity detection for which we have model and tutorial: https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb

Also we do have speaker recognition and verification models: https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb 

Great, thanks.  Voice Activity Detection is what I need.

A question about training: I have silence segments of varying sizes from very short to 10 or 20 seconds.  I also have speech of same distribution.  Is this correct for training?  Because I notice that the inference phase actually steps the net in very small increments like a 10th of a second with a window of half a second across the input.  I'm not sure how the training intersects with inference, i.e. should I really be training with a ton of 10th of a second samples or with the larger samples?

@fayejf FYI

Hi @catskillsresearch if I understand correctly, you are asking whether you should train on "half a second" or larger samples (10s)?

We train on segment-level (seg_len in [training notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb) and the default is 0.63s. For your 10 or 20 seconds input, we have a script (you can find it inside training notebook ) to help you to convert it to segment-level input
And in [inference phase](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb),  we shift the window (segment) by 10ms to generate the prediction of each frame (10ms).

Please let me know if you have any other questions.

Thanks, I will apply that script.  A few more questions:  The function `offline_inference` supplied in the 07_Online_Offline notebook moves the model to CPU for inference.  Can you give me a Python snippet that shows me how to take a NumPy array with 16K sample rate audio in it and move it to GPU and apply the model on GPU rather than CPU?  As-is, this model is extremely slow.

I have to do silence detection on a 10 minute clip.  For my purposes (single speaker speech detection), the following fairly simple gist totally nails it and is extremely fast:

```
T=np.arange(audio.shape[0])/sample_rate
S = librosa.feature.melspectrogram(y=audio, sr=sample_rate, n_mels=64, fmax=8000)
S_dB = librosa.power_to_db(S, ref=np.max)
s_dB_mean=np.mean(S_dB,axis=0)
speech_q=(s_dB_mean>-70).astype(float)
TSQ=T[-1]*np.arange(len(speech_q))/len(speech_q)

plt.figure(figsize=(60,8))
ax1 = plt.subplot(2,1,1)
ax1.plot(T, audio, color='blue');
ax1.plot(TSQ, 2*speech_q-1, color='red')
ax2 = plt.subplot(2,1,2)
librosa.display.specshow(S_dB, x_axis='time', y_axis='mel', sr=C.sample_rate, fmax=8000);
ax2.sharex(ax1)
plt.show()
```

@catskillsresearch  Sorry for late response.

The notebook is designed for demonstration and to be compatible with the various environments. You can find scripts in [PR #1384](https://github.com/NVIDIA/NeMo/pull/1384/) to assist you to perform 
**AUDIO -> MANIFEST -> VAD OUTPUT TXT -> VAD RTTM-like TABLE**

I'm closing this issue and please reopen or open a new one with questions.


> We don't have (yet) speaker diarization models yet.
> 
> This task: "returns a list of start and end indices in the audio array for each period of detected speech" sounds like voice activity detection for which we have model and tutorial: https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb

@okuchaiev How can I apply the Voice Activity Detection model to solve that problem? I followed the tutorial and was able to get the logits for my .wav file, but I don't know what to do now. I feel like I am using it incorrectly?

<details>
  <summary>Code</summary>

```
import nemo
import nemo.collections.asr as nemo_asr
from omegaconf import OmegaConf
import os
import pytorch_lightning as pl
import torch
import pickle

data_dir = 'data'
NEMO_ROOT = '.'

is_windows = os.name == 'nt'

test_manifest = os.path.join(data_dir, 'an4/wav/an4_clstk/dev.json')
hasan_manifest = os.path.join(data_dir, 'C:/Users/serg06/Downloads/v03/data/dataset.json')

manifest = hasan_manifest

def main():
    test_config = OmegaConf.create(dict(
        manifest_filepath = manifest,
        sample_rate = 22050,
        labels = None,
        batch_size = 1,
        shuffle = False,
        time_length = 8,
        embedding_dir = './'
    ))

    vad_model = nemo_asr.models.EncDecClassificationModel.restore_from('MatchboxNet_VAD_3x2.nemo')
    vad_model.setup_test_data(test_config)
    test_dl = vad_model._test_dl

    logits, labels = extract_logits(vad_model, test_dl)
    print(logits)
    print(labels)


@torch.no_grad()
def extract_logits(model, dataloader):
    logits_buffer = []
    label_buffer = []

    # Follow the above definition of the test_step
    i = 0
    for batch in dataloader:
        audio_signal, audio_signal_len, labels, labels_len = batch
        audio_signal, audio_signal_len, labels, labels_len = audio_signal.cuda(), audio_signal_len.cuda(), labels.cuda(), labels_len.cuda()

        logits = model(input_signal=audio_signal, input_signal_length=audio_signal_len)

        logits_buffer.append(logits)
        label_buffer.append(labels)

        print(".", end='')
    print()

    print("Finished extracting logits !")
    logits = torch.cat(logits_buffer, 0)
    labels = torch.cat(label_buffer, 0)
    return logits, labels


if __name__ == '__main__':
    main()
```
</details>

> Also we do have speaker recognition and verification models: https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb

I followed the tutorial for this one too, but I am bit confused on how I could use it to perform diarization. Would I have to split the audio on silence, then calculate speaker embeddings for each clip, then perform some kind of clustering on the embeddings, then construct the diarization timeline from all those clips + silences?

Also, if I'm using Speaker Recognition, does that mean that I can't detect when multiple people are speaking in the same audio clip?