Is there a way to deal with a known number of speakers?
First, thanks for this amazing works. This new diarization engine works like a charm.
Is there a way to improve the diariation when the number of speakers is known?
Also, I wonder if there are some good tricks to use that model when there are more than 8 speakers (other than having a fallback with another diarization engine when 8 speakers are detected by nemotron-diarization).
Hi @Jeronymous , thank you for your feedback and kind words!
Regarding your questions:
- Known number of speakers: At the moment, a specific mode that utilizes a predefined number of speakers (under 8) is not implemented.
- More than 8 speakers: We recommend looking into the EEND-VC approach. A good workaround is to perform local diarization in smaller windows using our model (keeping it up to 8 speakers per window), and then stitch those windows together using speaker embeddings.
Hope this helps!
Hi @Jeronymous , thank you for your feedback and kind words!
Regarding your questions:
- Known number of speakers: At the moment, a specific mode that utilizes a predefined number of speakers (under 8) is not implemented.
- More than 8 speakers: We recommend looking into the EEND-VC approach. A good workaround is to perform local diarization in smaller windows using our model (keeping it up to 8 speakers per window), and then stitch those windows together using speaker embeddings.
Hope this helps!
Hi! @imedennikov I noticed that you mentioned EEND-VC, using short windows and then exact embeddings to merge them. How about the speaker sync issue in real application? The speaker 1 in a window perhaps not the global speaker 1, so I want to know if you would like to share a little more about this with me thx.
Hi @Bohemian-self , sure! You’re right that local speaker labels are not consistent across windows. That’s exactly the problem EEND-VC solves.
EEND-VC in a nutshell: The recording is split into blocks, and an EEND model performs local diarization in each block. A speaker embedding is then extracted for every local speaker. All embeddings from all blocks are clustered globally with a cannot-link constraint, meaning two speakers from the same block can never be assigned to the same cluster. The cluster index becomes the global speaker ID. For details, see the original paper and its follow-up, which covers more robust constrained clustering.
With Nemotron-3-Diarization, a simple strategy would be:
- Run streaming inference until the model has found 8 speakers.
- Reset the diarization instance and start a new inference session from that point. Repeat until the end of the recording. Each session acts as one “window” with up to 8 speakers, and local IDs are consistent within it.
- For each speaker in each session, extract an embedding from their non-overlapped speech. You can use, e.g., TitaNet, WeSpeaker ResNet34 or ReDimNet2. Skip very short segments, since they give unreliable embeddings.
- Cluster all embeddings with the cannot-link constraint and map each local ID to its global one.
Good luck with your experiments!
Hi @Bohemian-self , sure! You’re right that local speaker labels are not consistent across windows. That’s exactly the problem EEND-VC solves.
EEND-VC in a nutshell: The recording is split into blocks, and an EEND model performs local diarization in each block. A speaker embedding is then extracted for every local speaker. All embeddings from all blocks are clustered globally with a cannot-link constraint, meaning two speakers from the same block can never be assigned to the same cluster. The cluster index becomes the global speaker ID. For details, see the original paper and its follow-up, which covers more robust constrained clustering.
With Nemotron-3-Diarization, a simple strategy would be:
- Run streaming inference until the model has found 8 speakers.
- Reset the diarization instance and start a new inference session from that point. Repeat until the end of the recording. Each session acts as one “window” with up to 8 speakers, and local IDs are consistent within it.
- For each speaker in each session, extract an embedding from their non-overlapped speech. You can use, e.g., TitaNet, WeSpeaker ResNet34 or ReDimNet2. Skip very short segments, since they give unreliable embeddings.
- Cluster all embeddings with the cannot-link constraint and map each local ID to its global one.
Good luck with your experiments!
Thanks for your reply and answering for my question! It's very kind of you. I will search for infos about your recommended way to make experiments. In fact that I have ran a exist pipeline powered by Nemotron-3-Diarization and ERes2Net, the performance of Nemotron-3-Diarization regarding both accuracy and speed are very impressive. On NVIDIA T4, it could reach even an RTF of 0.0034, and could clip short and quick spk converts much better than pyannote segmentation 3.0.