smdesai/Nemotron-3-Diarization-CoreML

Nemotron 3 Diarization — Core ML

Core ML conversion of NVIDIA's Nemotron 3 Diarization streaming speaker-diarization model (up to eight speakers, 10 ms output frames) for iOS 18+ / macOS 15+. Converted from checkpoint revision a435e9867d79e789e90053f9b6d6834053af564a (Nemotron-3-Diarization.nemo, SHA-256 867c53f552998f772e5b5e5c082962ae85ee7ca5669c2bc17d7f615133d4e96d). No weights were retrained, quantized or pruned.

The model is split into two graphs that a host streaming loop calls once per chunk: a preencoder (Mel features → 80 ms frame embeddings) and an encoder/head (packed speaker cache + FIFO + chunk embeddings → speaker probabilities). The Arrival-Order Speaker Cache, FIFO and chunking logic of Streaming Sortformer run on the host and are not part of these graphs; see "Streaming loop" below.

Contents

Path	Description
pipeline.json	Recommended. FP16 preencoder + FP16 BC1S encoder/head
pipeline-fp16.json	FP16 preencoder + FP16 standard-layout encoder/head
pipeline-fp32.json	FP32 preencoder + FP32 encoder/head (numerical reference)
models/preencoder_{fp32,fp16}.mlpackage	Preencoder graphs
models/encoder_head_{fp32,fp16}.mlpackage	Encoder/head, standard layout
models/encoder_head_fp16_bc1s.mlpackage	Encoder/head, Neural Engine layout (channel-first BC1S, pointwise Conv2d projections, per-head attention)
frontend/	Mel frontend parameters, Hann window and Mel filterbank
learned_silence_embedding.f32le	Learned silence embedding (512 × float32) used to pad the speaker cache
conversion/	Per-graph conversion manifests (inputs, outputs, tool versions, gates)
SHA256SUMS	SHA-256 of every file

.f32le files are raw little-endian float32 in C order. The .mlpackage directories load with MLModel.compileModel(at:) / Xcode. Ship the compiled .mlmodelc: the BC1S encoder needs a ~35 s Neural Engine compile on first load, which the system caches for a compiled model at a stable path.

Streaming profile

All graphs are fixed-shape for the model card's "Low latency" (1.04 s) configuration, applied synchronously (cache update inside each step):

Parameter	Value
Chunk length	9 encoder frames (0.72 s)
Left / right context	1 / 4 encoder frames (0.08 / 0.32 s)
Speaker cache / FIFO	264 / 264 frames
Speaker-cache update period	222 frames
Encoder frame	8 Mel frames (80 ms)
Output resolution	10 ms (8× upsampled), 8 speakers

Other latency configurations need a re-export with different fixed shapes.

Frontend

16 kHz mono float32 audio → pre-emphasis 0.97 → centered STFT (FFT 512, Hann window 400, hop 160, constant zero padding) → power spectrum → 128 Slaney Mel bins → log(x + 2^-24). No dither, no feature normalization. floor(samples / 160) valid frames. The window and filterbank are the checkpoint's own buffers, which NVIDIA stores in bfloat16; they differ slightly from freshly computed float32 values. Project each frame's power spectrum onto the filterbank separately (a batched matrix product changes the float rounding and makes long streams diverge).

Graph interfaces

Preencoder

Name	Shape	Type
in mel_features	[1, 112, 128] time-major (left 8 + central 72 + right 32 frames, zero-padded)	float32
in mel_length	[1] valid frames	int32
out chunk_embeddings	[1, 14, 512]	float32
out chunk_embedding_length	[1] = ceil(mel_length / 8)	int32

Encoder/head (all variants)

Name	Shape	Type
in packed_embeddings	[1, 542, 512]: speaker cache, then FIFO, then chunk embeddings, zero-padded	float32
in packed_length	[1] valid packed rows	int32
out native_probabilities	[1, 4336, 8] speaker activity at 10 ms (542 × 8 rows)	float32
out cache_probabilities	[1, 542, 8] per packed row, for cache/FIFO updates	float32

Probabilities are sigmoid outputs; the model card's default decision threshold is 0.5.

Streaming loop

Per chunk: slice left/central/right Mel frames, run the preencoder, pack [speaker cache | FIFO | chunk embeddings] up to 542 rows, run the encoder/head, emit the central chunk's rows of native_probabilities, then update the FIFO and speaker cache from cache_probabilities following NVIDIA NeMo's Streaming Sortformer (Arrival-Order Speaker Cache compression every 222 frames, learned silence embedding for padding). On the final call, flush the remaining chunks with zero right context and emit only rows up to the true end of the audio.

Validation

Against the original NeMo FP32 model with the same streaming profile:

• FP32: strict agreement (absolute 2e-6 + relative 2e-5) on every output of 305 teacher-forced checks, a 30 s closed-loop stream (Mac CPU and iPhone 17 Pro) and a 120 s stress stream crossing five cache compressions (Mac CPU). Identical segments; 30 s sample DER 4.52% (zero collar, overlap included).
• FP16 (both encoder layouts): FP16 rounding exceeds the strict FP32 tolerance but the 30 s DER is identical to FP32 (4.52%), with 4 of 3,000 frames changing a speaker decision on iPhone.
• BC1S encoder: outputs byte-identical to the FP16 standard-layout encoder on the iPhone Neural Engine for 30 s, 376 s and 36 min recordings.

Speed, iPhone 17 Pro (iOS 27.2), CPU + Neural Engine, 36 min recording: BC1S encoder 102.9 s (20.8× real time, 34 ms per encoder call) vs standard FP16 133.6 s (16.0×, 44 ms); BC1S peak memory is about 40–60 MB higher. An Instruments trace showed every BC1S encoder call on the Neural Engine.

These are checks against the reference model on a few recordings, not a new accuracy benchmark; see NVIDIA's model card for accuracy.

License

Use is governed by the OpenMDW License Agreement, version 1.1 (see LICENSE), the license of the original model. See NOTICE for origin.