Question: what is the best approach to train speaker diarizer
State: closed | Pull request: no
Labels: name: stale

Hi, 

I need to use _speaker diarization_ in a pipeline with ASR for Slovenian language. For the speech recognition I use a highly accurate  `CTC Conformer`  model. 

At first, I tried to employ pre-trained `titanet-large` and `diar_msdd_telephonic` models for clustering and neural diarization, respectively. But results are not satisfactory. In particular not for audio files with multiple speakers, e.g. more than 10. For larger files I would often get one label only. And very often, in speech segments belonging to one speaker, single words would be misclassified. I experimented a lot with all sorts of parameter combinations and pre-set config files (`diar_infer_meeting`, `diar_infer_general`, ...) but no success.

Then I trained from scratch a new `titanet-large` model following instructions on [NeMo github](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb) and also finetuned one with my own dataset. The dataset includes 2500 unique speakers and approx. 700 hours of speech. Audio files are from 1 to 20 seconds long. Considering the loss function, both models converged well with the one trained from scratch ending up at `val_loss=0,13` and finetuned one at `0,00006`. When I use these two models for clustering diarization, however, results are still quite disappointing – short audio recordings with few speakers are diarized well but immediately when there are more than a few speakers, DER increases significantly. Results are sometimes even worse if I pre-set the number of speakers.

I also trained own MSDD model by following the tutorial “[Speaker Diarization Training](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Training.ipynb)". I tried both, training by freezing speaker embedding extractor model and by end-to-end training of both together. But still, I don't get any better results. In fact, there is almost no difference between the results I get by clustering diarization alone and by using neural diarization on the final step.

I guess I must be doing something wrong. Could somebody please let me know, what is the best approach to get this done. I need a diarizer that will perform well for recordings with just a few or multiple speakers, which will very often be unknown (I don't have them in my training dataset). Specifically:

1.	Should I train or rather finetune a speaker embedding extractor model (e.g. titanet-large) as suggested in [this tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb) and then employ clustering diarization without msdd? If so, is there anything else apart from what is instructed in the **Speaker Verification tutorial** that I must change or pay attention to to train speaker embedding extractor model for the diarization purposes (the tutorial is on verification not diarization)? Is the dataset I have (700 hours, 2500 unique speakers) good enough or I need more data. What is suggested duration for recordings?   

2.	Or should I rather train msdd with frozen speaker embeddings extractor or even end-to-end as suggested [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Training.ipynb) and then use neural diarization on top of the clustering results? Are there any specifics I need to know apart from what is explained in the Speaker Diarization Training tutorial?

If required, I can provide details on config settings that I was using for my previous experiments. 

Thanks in advance.