nikioth/titanet-large-coreml

TitaNet-L speaker embedding for Core ML

NVIDIA's TitaNet-L speaker verification model (nvidia/speakerverification_en_titanet_large) converted to a Core ML ML Program. It maps a log-mel spectrogram of one utterance to a 192-dimensional speaker embedding. Embeddings of the same voice have a high cosine similarity, so the model can enroll and recognize known speakers, verify that two clips share a speaker, or feed a clustering step.

The weights are NVIDIA's, unchanged in value and kept in float32. The network was rebuilt in plain PyTorch from the ONNX export that k2-fsa/sherpa-onnx distributes (nemo_en_titanet_large.onnx), checked against onnxruntime and converted with coremltools. It needs no NeMo, PyTorch or ONNX Runtime at inference time.

File	TitaNet.mlpackage (88.5 MB, float32 weights and compute)
Input	features, Float32, shape (1, 80, T), T = 50 to 12288 frames (0.5 s to about 2 min)
Output	embedding, Float32, shape (1, 192)
Deployment target	macOS 14 / iOS 17 or later
Compare with	cosine similarity

Files

TitaNet.mlpackage/                  the model (load this, or compile it to .mlmodelc)
example.py                          Python: compare two WAV files
swift/TitaNet.swift                 Swift: feature frontend + model call, one file
conversion/export_titanet_coreml.py the conversion script that produced the package
LICENSE                             CC BY 4.0 legal code

Input features

The model does not include its frontend. Feed it the same features sherpa-onnx computes for TitaNet, which are kaldi-native-fbank log-mel energies with these settings:

Setting	Value
Audio	mono, 16 kHz, float samples in [-1, 1]
Frames	25 ms (400 samples) every 10 ms (160 samples), snip_edges = true
Window	Hann (periodic: 0.5 - 0.5 cos(2πn/400))
Pre-emphasis	0.97 (Kaldi style, x[0] -= 0.97 x[0])
DC offset removal	off
Dither	0
FFT	512 points, power spectrum, Nyquist bin included (257 bins)
Mel filterbank	80 bins, 0 to 7600 Hz, librosa style (Slaney mel scale, Slaney area normalization)
Compression	natural log, floored at float epsilon
Normalization	per mel bin over the whole utterance: subtract the mean, divide by the population standard deviation + 1e-5

In kaldi-native-fbank terms: window_type="hann", remove_dc_offset=False, dither=0, preemph_coeff=0.97, num_bins=80, low_freq=0, high_freq=-400, is_librosa=True. Lay the result out as (1, 80, frames), mel bins on the second axis.

Usage

Python (coremltools, macOS)

import coremltools as ct
import kaldi_native_fbank as knf
import numpy as np
import soundfile as sf

def features(samples):
    o = knf.FbankOptions()
    o.frame_opts.samp_freq = 16000
    o.frame_opts.snip_edges = True
    o.frame_opts.dither = 0
    o.frame_opts.remove_dc_offset = False
    o.frame_opts.preemph_coeff = 0.97
    o.frame_opts.window_type = "hann"
    o.mel_opts.num_bins = 80
    o.mel_opts.low_freq = 0
    o.mel_opts.high_freq = -400
    o.mel_opts.is_librosa = True
    fb = knf.OnlineFbank(o)
    fb.accept_waveform(16000, samples.tolist())
    fb.input_finished()
    f = np.array([fb.get_frame(i) for i in range(fb.num_frames_ready)], dtype=np.float32)
    f = (f - f.mean(0)) / (f.std(0) + 1e-5)
    return f.T[None].copy()          # (1, 80, T)

model = ct.models.MLModel("TitaNet.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)
a = model.predict({"features": features(sf.read("a.wav", dtype="float32")[0])})["embedding"][0]
b = model.predict({"features": features(sf.read("b.wav", dtype="float32")[0])})["embedding"][0]
print(a @ b / np.linalg.norm(a) / np.linalg.norm(b))

example.py is the same code as a script (pip install coremltools kaldi-native-fbank soundfile numpy, then python example.py a.wav b.wav).

Swift (Core ML)

swift/TitaNet.swift is a self-contained file with the frontend (Accelerate) and the model call:

let model = try TitaNet(modelURL: URL(fileURLWithPath: "TitaNet.mlpackage"))
let a = try model.embedding(of: samplesA)   // [Float], mono 16 kHz in [-1, 1]
let b = try model.embedding(of: samplesB)
print(TitaNet.cosine(a, b))

Its embeddings match the Python path above at cosine 1.0000 on the test clips below.

Parity with the original

The conversion script compares Core ML with onnxruntime running the sherpa-onnx file (SHA-256 d51abcf3...daca8e77) on random features of 50 to 3000 frames:

Compute units	Worst cosine vs onnxruntime
CPU only	1.000000
CPU and GPU	1.000000
All (CPU, GPU, Neural Engine)	1.000000

Rebuilt PyTorch vs onnxruntime is also 1.0000000 at 150, 333, 600 and 1200 frames. On three synthetic speech clips (two different macOS voices, one of them saying two different sentences) the features above give cosine 1.0 against onnxruntime on every compute unit; the same voice scores 0.897 and two voices saying the same sentence score 0.306.

A float16 conversion was tried and rejected: its worst cosine against onnxruntime was 0.9988, a visible loss for a model whose decisions sit on cosine thresholds, so the package keeps float32. On Apple Silicon float32 runs on the CPU and GPU; the Neural Engine is not used.

Speaker identification on real calls

TitaNet-L was chosen for this conversion by an evaluation on a private set of 19 recorded multi-speaker calls with 16 known people (57 labeled voices). Test windows were 1.5 to 6 s of clean single-speaker speech (no other speaker within 0.25 s, at least 4 words, at least 80% speech), 7,030 windows in all. Scoring was leave-one-meeting-out: each person's windows from one call (46 test units) were matched against their samples from the other calls. Every model was run as ONNX on the CPU, so the numbers below are the source models'; the Core ML package reproduces TitaNet-L's embeddings at cosine 1.000000.

Model (license)	Rank-1, per voice	EER, per voice	One window: rank-1 / EER	ms per window (ONNX, CPU)
WeSpeaker CAM++ (CC BY 4.0)	30/46	13-19%	27% / 58%	21
WeSpeaker ResNet34-LM + PLDA (CC BY 4.0)	34/46	9.1%	80% / 25%	46
WeSpeaker ResNet221-LM (CC BY 4.0)	36/46	10-15%	49% / 52%	304
3D-Speaker CAM++ zh+en advanced (Apache-2.0)	44/46	4.4%	97.5% / 6.8%	21
3D-Speaker ERes2Net VoxCeleb (Apache-2.0)	44/46	4.4%	97.9% / 7.1%	75
NVIDIA TitaNet-S (CC BY 4.0)	44/46	4.4%	96.7% / 10.0%	24
NVIDIA TitaNet-L (CC BY 4.0)	44/46	4.4%	97.6% / 5.1%	55

The strongest models all reach 44 of 46 per voice (the two misses are voices with one and two windows). On single windows TitaNet-L has the lowest equal error rate, 5.1% overall and 6.4% on windows of 3 s or less. Part of the remaining error is probably label noise: every strong model hears two of the labeled people as one voice.

Speed

Median latency of one prediction of TitaNet.mlpackage with coremltools on an Apple M2 Pro (32 GB), macOS 27, after warm-up:

Window	CPU only	CPU and GPU	All
1.5 s (148 frames)	17.3 ms	4.4 ms	4.5 ms
3 s (298 frames)	32.5 ms	6.5 ms	6.6 ms
6 s (598 frames)	62.8 ms	9.6 ms	10.9 ms

Limitations

• TitaNet-L was trained on English speech (VoxCeleb 1 and 2, Fisher, Switchboard, LibriSpeech, NIST SRE 2004-2010). Other languages and channels may need their own thresholds.
• The call-audio figures above come from one small private set (19 calls, 16 people, recorded call audio). Treat them as an indication and tune any threshold on your own data.
• The evaluation used windows of 1.5 s and longer; shorter clips were not measured. The model refuses input under 0.5 s (50 frames).
• One utterance per call: there is no batch dimension or length mask, so pad nothing and call the model once per clip.
• This is not a biometric authentication system. It has no spoofing or liveness protection and should not gate access to anything.

Reproducing the conversion

pip install torch "coremltools>=8" onnx onnxruntime numpy
# nemo_en_titanet_large.onnx from
# https://github.com/k2-fsa/sherpa-onnx/releases/tag/speaker-recongition-models
python conversion/export_titanet_coreml.py --onnx nemo_en_titanet_large.onnx --out out/

The script rebuilds TitaNet-L in PyTorch from the ONNX initializers, prints PyTorch vs onnxruntime and Core ML vs onnxruntime cosines, and writes out/TitaNet.mlpackage. With torch 2.15 and coremltools 9.0 it reproduces this repository's weight.bin byte for byte.

License and attribution

The model is NVIDIA's TitaNet-L from NeMo (titanet_large), released under CC BY 4.0; this conversion keeps that license (LICENSE). The ONNX export it was converted from is distributed by k2-fsa/sherpa-onnx. Changes made by nikioth: the network was rebuilt in PyTorch from the ONNX weights and converted to Core ML (float32, flexible frame count); the weights were not retrained or modified. The scripts in this repository are provided under the same license. Provided as is, without warranties of any kind.

Citation

@inproceedings{koluguri2022titanet,
  title     = {TitaNet: Neural Model for Speaker Representation with 1D Depth-wise
               Separable Convolutions and Global Context},
  author    = {Koluguri, Nithin Rao and Park, Taejin and Ginsburg, Boris},
  booktitle = {ICASSP 2022 - IEEE International Conference on Acoustics, Speech
               and Signal Processing},
  pages     = {8102--8106},
  year      = {2022}
}