FlaxSpeaker: vox_ecapa_tdnn_small_ext_softmax (ecapa_tdnn - small)

Disclaimer & Reproduction Notice

This is not an officially supported Google product.

This checkpoint and the accompanying FlaxSpeaker library are a from-scratch re-implementation in JAX/Flax, written using only the published papers (no proprietary code or data), of speaker recognition algorithms from:

• GE2E: Generalized End-to-End Loss for Speaker Verification (arXiv:1710.10467)
• Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognition (arXiv:2104.01989)
• ATP (pooling head repurposed from a language identification paper): Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech (arXiv:2202.12163)
• PFAS: Parameter-Free Attentive Scoring for Speaker Verification (arXiv:2203.05642)

The experiments in those papers were run on Google-internal infrastructure and data, not with this library. This is a 400-step demonstration checkpoint, not a reproduction of the papers' results -- see Training budget and Deviations from the source papers below.

Training Budget (read this first)

This model was trained for 400 gradient steps with mini-batches of 32 utterances (8 speakers x 4 utterances), i.e. about 1.35 epochs over the 9,447-utterance universal corpus, on a CPU workstation, with a single seed (42). Losses are not converged at this budget. All EERs below are therefore far from state of the art and are only meaningful for relative comparisons between FlaxSpeaker checkpoints trained under the same budget.

Training-crop provenance. This checkpoint was trained with fixed 180-frame (1.8 s) crops and all-ones validity masks, not with the U(90, 180) variable-length crops that are the current pipeline default: the training code in use at the time defaulted the batchers to variable_length=False and the benchmark driver did not pass the flag. The shipped config.json / YAML therefore set variable_length_training: false, which re-creates this regime. Step-0 loss fingerprint at the released seed (scripts/fingerprint_released_checkpoints.py, pretrained_models/step0_fingerprints.json): archived 2.0428; current code with fixed 180-frame crops 2.0428, with U(90, 180) crops 2.0999 (match: fixed). Evaluating the released weights with the current code reproduces the archived metrics in eval_results.json. The seed-to-seed standard deviation of pooled 1-utt EER at this budget is about 0.7-0.8 % absolute, so sub-1.5 % differences between checkpoints are not rankings.

For orientation only: published ECAPA-TDNN systems trained on VoxCeleb2-dev reach about 0.9% EER on the full VoxCeleb1-O list (Desplanques et al., 2020), and sub-1M-parameter systems about 2% (Wang et al., EfficientTDNN, 2022). The 400-step checkpoints released here are 10-30x worse than that and are intended for relative comparisons within the FlaxSpeaker benchmark, not as state-of-the-art models.

Model Summary

Property	Value
Model ID	vox_ecapa_tdnn_small_ext_softmax
Hugging Face Repository	wq2012/vox_ecapa_tdnn_small_ext_softmax
Hugging Face Collection	FlaxSpeaker Collection
GitHub Repository	wq2012/FlaxSpeaker
PyPI Package	flaxspeaker
Backbone Architecture	ecapa_tdnn (small)
Temporal Pooling	asp
Training Objective	Blended Extended-Set Softmax (FlaxSpeaker variant: 0.50 row-softmax + 0.50 extended-set with a -log N logit shift; not the paper's Eq. 2) (extended_set_softmax)
Verification Scorer	Cosine similarity (cosine)
Output Representation Dim	192
Inference Parameters (encoder + scoring head; excludes training-only classifier weights)	1,684,418
Training Data	VoxCeleb1 only (30 speakers from the VoxCeleb1 test partition)
Training Steps / Batch	400 steps x 32 utterances, seed 42
Audio Frontend	80-dim natural-log Mel @ 16000 Hz + frame-level energy VAD + CMVN (flaxspeaker.AudioFrontend; not a Hugging Face AutoFeatureExtractor)

Model Files Included

• model.safetensors: model weights (encoder + GE2E w,b / Dr-Vector / PFAS heads) in SafeTensors format
• flax_model.msgpack: native Flax Msgpack serialized parameters
• config.json: FlaxSpeakerConfig (full ExperimentConfig, incl. the frontend/VAD settings)
• eval_results.json: multi-domain evaluation metrics (1-utt & 3-utt EER, minDCF, ROC-AUC, 95% bootstrap CIs, VoxCeleb1-O 10-speaker subset)

Quickstart Usage

Python API (from Hugging Face Hub or Local Directory)

Requires flaxspeaker >= 0.3.0 (pip install -U flaxspeaker); the 0.2.0 wheel on PyPI predates the released checkpoints and cannot load them.

from flaxspeaker import load_pretrained

model = load_pretrained("wq2012/vox_ecapa_tdnn_small_ext_softmax")
emb = model.extract_embedding("path/to/utterance.flac")
score = model.verify("path/to/enroll.flac", "path/to/test.flac")
print(f"Embedding shape: {emb.shape}, Verification score: {score:.4f}")

Command-Line Inference

python scripts/inference.py --model_dir wq2012/vox_ecapa_tdnn_small_ext_softmax \
    --enroll_audio path/to/enroll.flac --test_audio path/to/test.flac

Multi-Domain Evaluation Results

1-Utterance Verification Across Evaluation Domains

Trial lists are 50/50 target/impostor over held-out speakers (LibriSpeech 40, CN-Celeb2 30, VoxCeleb1 10 speakers); minDCF@0.01 is close to its trivial ceiling of 1.0 for every model under this trial mix.

Evaluation Split	Trials	EER (%)	95% Bootstrap CI (%)	minDCF@0.01	minDCF@0.10	ROC-AUC
LibriSpeech (English Read)	3,000	20.07%	[18.52, 21.55]	0.9860	0.9553	0.8815
CN-Celeb2 (Chinese Multi-Genre)	3,000	32.40%	[30.58, 34.05]	0.9973	0.9840	0.7473
VoxCeleb1 (In-the-Wild Celebrity)	2,000	21.10%	[19.30, 23.30]	0.9940	0.9510	0.8690
Universal Pooled (All 3 Domains)	8,000	24.57%	[23.73, 25.48]	0.9952	0.9700	0.8328
Universal Macro-Average (3 Domains)	—	24.52%	—	0.9924	0.9634	0.8326
VoxCeleb1-O 10-speaker subset (see caveat)	2,500	22.52%	[20.50, 24.38]	0.9899	0.8816	0.8627

Caveat: VoxCeleb1-O 10-speaker subset: the official veri_test2.txt list filtered to trials whose enrollment and test speakers are both among the 10 held-out evaluation speakers (2,500 trials, ~72% target). Not comparable to published VoxCeleb1-O numbers; the other 30 of the 40 official test speakers are used as training data by vox_* and univ_* models.

3-Utterance Multi-Enrollment Verification Across Evaluation Domains

Evaluation Split	Trials	3-Utt EER (%)	95% Bootstrap CI (%)	minDCF@0.01	ROC-AUC
LibriSpeech (3-Utt)	1,000	14.60%	[12.40, 17.05]	0.9560	0.9242
CN-Celeb2 (3-Utt)	1,000	30.60%	[27.80, 33.35]	0.9760	0.7659
VoxCeleb1 (3-Utt)	1,000	17.80%	[14.80, 20.50]	0.9920	0.9050
Universal Pooled (3-Utt)	3,000	20.97%	[19.57, 22.57]	0.9907	0.8730
Universal Macro-Average (3-Utt)	—	21.00%	—	0.9747	0.8650

Single seed; the 95% bootstrap CIs resample trials (not speakers). Differences below about 1% absolute EER between FlaxSpeaker checkpoints are within the seed-to-seed variation measured in pretrained_models/r2_seed_and_control_results.json (see the GitHub README) and should not be interpreted as rankings.

Deviations from the Source Papers

• Training regime: 400 gradient steps x 32 utterances (N = 8 speakers x M = 4) on <= 25 utterances/speaker from 381 (universal) or fewer speakers; the source papers train for ~500k steps with N = 16-64 / M = 8-10 on thousands of speakers.
• Augmentation: synthetic log-Mel-domain noise + one-tap smoothing (specaug.apply_feature_mtr_augmentation), not waveform MTR with recorded noise/RIR corpora.
• Objective: 50/50 blend of row-softmax and -log N-shifted extended-set softmax; the paper's pure Eq. 2 is available via pure_extended_set: true.
• ECAPA-TDNN: LayerNorm instead of BatchNorm.
• Frontend: 80-D natural-log Mel + frame-level energy VAD (26 dB dynamic range) + CMVN; evaluation centre-crops to at most 500 frames (5 s).

Citation

@inproceedings{wan2018generalized,
  title={Generalized End-to-End Loss for Speaker Verification},
  author={Wan, Li and Wang, Quan and Papir, Alan and Moreno, Ignacio Lopez},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={4879--4883},
  year={2018}
}

@inproceedings{pelecanos2021dr,
  title={Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognition},
  author={Pelecanos, Jason and Wang, Quan and Moreno, Ignacio Lopez},
  booktitle={Interspeech},
  pages={4603--4607},
  year={2021}
}

@inproceedings{wang2022attentive,
  title={Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech},
  author={Wang, Quan and Yu, Yang and Pelecanos, Jason and Huang, Yiling and Moreno, Ignacio Lopez},
  booktitle={The Speaker and Language Recognition Workshop (Odyssey)},
  year={2022}
}

@inproceedings{pelecanos2022parameter,
  title={Parameter-Free Attentive Scoring for Speaker Verification},
  author={Pelecanos, Jason and Wang, Quan and Huang, Yiling and Moreno, Ignacio Lopez},
  booktitle={The Speaker and Language Recognition Workshop (Odyssey)},
  year={2022}
}