Speaker Diarization - Extracting speaker embeddings for labels from files with multiple speakers from Cluster Diarizer models
State: closed | Pull request: no
Labels: name: stale

Hello,

I'm currently engaged in diarization for meetings domains using the **ClusteringDiarizer** model from the pre-trained pipeline, along with the meeting configuration (_diar_infer_meeting.yaml_), which has proven quite effective for my use case, just relying on listening by ear. Thanks for creating such a great library!  My goal is to acquire cluster-averaged speaker embeddings for each speaker based on the clustering results. However, I'm facing challenges in finding a direct method to retrieve these embeddings from the **ClusteringDiarizer** module.

I came across a related issue (#6965) for **EncDecDiarLabelModel** (MSDD) models, where it was recommended to obtain the [ms_avg_embs](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L417C24-L417C79) variable from the [MSDD forward function](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L565), which utilizes the _get_cluster_avg_embs_model_ function.

### Problem with MSDD model
Thus, I tried to use the MSDD model instead of the ClusteringDiarizer model instead. However, I am facing some challenges as the MSDD model for the meeting domain has not been released (#6748). When attempting to load the MSDD model with the meeting configuration (diar_infer_meeting.yaml) and the "diar_msdd_telephonic" model path, I encounter a size mismatch error after running diarize(). This is likely due to differences in parameters and configurations between domains. I could train a MSDD model on a meeting domain corpus with the correct configurations but that would take some time and effort

### Workaround with get_cluster_avg_embs method
As a workaround, I employed the following approach:

```
msdd_model.clustering_embedding.prepare_cluster_embs_infer()
msdd_model.clustering_embedding.emb_sess_test_dict
```
I opted for the [_prepare_cluster_embs_infer()_](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L694C1-L701C10) method, which internally calls the [_run_clustering_diarizer()_](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L825C14-L825C14) method, and, in turn, uses the [**_get_cluster_avg_embs_**](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L764) method. I then accessed msdd_model.clustering_embedding.emb_sess_test_dict to retrieve cluster average embeddings. Essentially, I am running the clustering diarizer step with the MSDD model without feeding the output into the neural diarizer model.

I have 3 questions: 

### Question 1: get_cluster_avg_embs vs get_cluster_avg_embs_model method: Correct Approach?
I would like your thoughts on this approach for obtaining cluster average embeddings and whether it is the correct strategy. I'm unclear about the distinctions between [**_get_cluster_avg_embs_**](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L764) and [**_get_cluster_avg_embs_model_**](https://github.com/NVIDIA/NeMo/blob/5a5c77aab6a2568987a19ad78299c3c57bcc9766/nemo/collections/asr/models/msdd_models.py#L384). The former is used by prepare_cluster_embs_infer(), and the latter is used in the MSDD forward function. Do they essentially return the same information? The documentation mentions that **_get_cluster_avg_embs_** returns _emb_sess_avg_dict_, a dictionary containing speaker mapping information and cluster-average speaker embedding vectors. The emb_sess_avg_dict keys appear to correspond to the scales (0, 1, 2, etc.), and the associated values are embeddings with a shape of (192, num_of_speakers). I'm also little confused by why  the number of identified speakers in the prediction RTTM doesn't match the specified num_of_speakers, perhaps you can provide some insight into this!

### Question 2: Obtaining cluster averaged embeddings from the ClusteringDiarizer model instead of MSDD
I'm also curious if there's a more direct way to retrieve embeddings from the clustering diarizer module without relying on this workaround with the MSDD model. The outcomes from the clustering diarizer module, using both diarize() and the MSDD model's .prepare_cluster_embs_infer(), exhibit slight differences (4 vs 5 speakers, the statistics were roughly the same but it seemed that two clusters got merged where one of the cluster was very small in size), which is quite confusing to me as I expected them to utilize the same methods. The MSDD model also seems to run much slower then the ClusteringDiarizer Model.

### Question 3: Best representation of embeddings for each speaker
Lastly, I'm seeking guidance on obtaining the most reliable representation of the representative embeddings for each speaker, considering multiple scales. My assumption is that a straightforward approach would be to calculate the average of all the embeddings for each scale. But it is mentioned that decisions are made ultimately on the base scale....if I would like to compare embeddings for speaker recognition or visualise them on a plot, is average or taking the base scale better?

Thank you once again!

Zack

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

the same question

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Hi @sunraymoonbeam @maxpain 
Sorry for missing this question and not answering this in timely manner. 
Multi-scale diarization decoder (MSDD) model only supports `diar_infer_telephonic.yaml`.
So you cannot use `diar_infer_meeting.yaml` settings.

- Question 1
get_cluster_avg_embs_model() function is as the name suggests, it is designed for model training.
get_cluster_avg_embs() is for inference case, so for your purpose I think using `get_cluster_avg_embs()` function makes more sense. 

- Question 2
MSDD inference requires the Clustering Diarizer results, so MSDD inference takes more time than clustering diarizer.
Also, you will get different results depending on scale length, scale weights when you run clustering diarizer. 
Check if you use the same Titanet model, the same scale lengths and scale weights. Then it would generate the same results.

- Question 3 
If you want to use speaker embeddings for speaker verification, definitely go with the longer scales.
However, to get the best results, extract TitaNet speaker embeddings from 5\~10 seconds of audio.
I recommend you to avoid using the averaged embeddings from diarization process (these are two short),
and I suggest to use TitaNet stand alone model to generate speaker embedding from 5~10 seconds of audio from the final diarization results.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

As @sunraymoonbeam mentioned  get_cluster_avg_embs returns emb_sess_avg_dict, a dictionary containing speaker mapping information and cluster-average speaker embedding vectors. The emb_sess_avg_dict keys appear to correspond to the scales (0, 1, 2, etc.), and the associated values are embeddings with a shape of (192, num_of_speakers). I'm also little confused by why the number of identified speakers in the prediction RTTM doesn't match the specified num_of_speakers, perhaps you can provide some insight into this!

I am trying to find a match with my own speaker embeddings so I can assign a "speaker" to that embeddings. Why does the identified speakers in the prediction RTTM not match the specified num_of_speakers?