Speaker diarization models for Whiscribe (ONNX)
The two ONNX models Whiscribe, an Android app that transcribes audio on the phone, uses to find who speaks when. They run on ONNX Runtime's CPU provider, with a small pipeline of the app's own around them.
| File | Size | SHA-256 | What it does |
|---|---|---|---|
pyannote-segmentation-3.0.onnx |
5,992,913 bytes | 220ad67ca923bef2fa91f2390c786097bf305bceb5e261d4af67b38e938e1079 |
Who talks in each 10 s window, up to three speakers (powerset output) |
redimnet2-b3-vox2-lm.onnx |
18,041,088 bytes | 4778b1e84c49d7e3dc0ee5caf6aef69a357f42a8f73d38b42cb39bf39024e452 |
A voice as 192 numbers (speaker embedding), from 16 kHz audio |
Segmentation: input x float32 [batch, 1, 160000] (10 s at 16 kHz), output y [batch, 589, 7]: per frame, the
scores of the 7 powerset classes (nobody, speaker 1, 2, 3, 1+2, 1+3, 2+3).
Embedding: input wav float32 [1, samples] at 16 kHz, mono, in [-1, 1], any length (the app uses 0.5 s or more);
output emb [1, 192]. Compare two by cosine similarity.
How Whiscribe uses them
pyannote segmentation on 10 s windows every 2.5 s; one embedding per window speaker, from the audio where only they talk; average-linkage clustering on cosine distance, cut at 0.6; small clusters merged into the nearest bigger one in time. Measured on AMI meetings and made-up conversations of real voices in eight languages: see the app's design notes.
Credits and licence
- pyannote segmentation 3.0 by pyannote (CNRS), MIT
(
LICENSE-pyannote). This is the ONNX conversion published by k2-fsa/sherpa-onnx (sherpa-onnx-pyannote-segmentation-3-0.tar.bz2, itsmodel.onnx), unchanged. - ReDimNet2-B3 by Palabra.ai, MIT (
LICENSE-redimnet2): the weightsb3-vox2-lm.ptfrom its v1.0.0 release (SHA-25602d14efdc184573873b5c3f62bea847fcdb6919fbba19a8da7af7c1ba36b76a2), trained on VoxCeleb2 only, exported to ONNX at opset 17 withexport_redimnet2_onnx.py(repository at commitc5bbe0b76e37df698c403f8844e41304ceab6307, PyTorch's TorchScript exporter). Its embeddings match PyTorch's (cosine 1.000000 on 0.5–10 s of audio). Nothing was retrained.
This repository is MIT-licensed as well. It isn't affiliated with or endorsed by pyannote, CNRS, k2-fsa or Palabra.ai.
About the training data: VoxCeleb2's audio comes from YouTube interviews; its metadata is CC BY-SA 4.0 and the videos' copyright stays with their owners, the same lineage as nearly every English speaker model. ReDimNet2's weights trained with VoxBlink2 are CC BY-NC-SA and are not used here.
python export_redimnet2_onnx.py path/to/redimnet2 b3-vox2-lm.pt redimnet2-b3-vox2-lm.onnx