lazarenkod/sortformer-ru-streaming-coreml

Streaming Sortformer, fine-tuned on Russian simulated meetings (Core ML)

Licensed by NVIDIA Corporation under the NVIDIA Open Model License. This is a modified version of nvidia/diar_streaming_sortformer_4spk-v2.1 (four speaker slots, about one second of output latency), fine-tuned and converted to Core ML for the Promises app on Apple Silicon. The LICENSE file is the agreement; it governs this model. The Core ML conversion follows FluidInference/mobius (Apache-2.0).

SortformerPipeline.mlmodelc is the compiled pipeline (mel features, speaker cache and FIFO in; speaker probabilities out), configuration "balanced v2.1" (chunk 6, left context 1, right context 7, FIFO 188, speaker cache 188, update period 144), to be driven by the streaming loop of FluidAudio (SortformerModels(config: .balancedV2_1, main: MLModel)).

What was changed

54 steps (batch 12 x 90 s, lr 5e-5, bf16, 75 min), the model's own streaming pass and loss (NeMo), on meetings simulated on the fly from Common Voice Russian speakers (497 voices of the training part, 16 h): 2 to 4 speakers per 90 s segment, overlap 10 to 35 %, room impulse responses of OpenSLR 28, MUSAN noise, 15 % telephone band. Against forgetting, 25 % of the training segments are windows of real meetings: AMI training meetings (AMI Meeting Corpus, CC BY 4.0, https://groups.inf.ed.ac.uk/ami/) and VoxConverse test recordings (Chung et al., "Spot the conversation: speaker diarisation in the wild", Interspeech 2020, arXiv:2007.01216). The meetings of the evaluation sets below were not used for training; the simulated evaluation voices are disjoint from the training voices.

Measured (DER, collar 0.25, overlap scored, one Core ML run each)

Set	Original v2.1 (same converter)	This model
Russian simulation, 15 meetings	15.6	11.0
Russian simulation, 30 meetings (dev voices)	17.9	13.7
VoxConverse, 14 recordings	9.1	9.7
AMI EN2002b / ES2004a	33.1 / 28.8	29.0 / 25.5

Limits (read before relying on it)

• The Russian gain is measured on simulated meetings built from the same kind of voices as the training ones (disjoint speakers); it is probably an overstatement for real Russian meetings. No real Russian meeting has been scored.
• More than four speakers, meetings longer than five minutes, and the model's latency and energy use on a live call were not measured.
• Each number is one run; AMI ES2004b swings by more than 10 points between runs of the same model.
• Four speaker slots, as in the original.