DISENT-KWS: Speech Disentanglement for Robust Custom Word Detection

Overview

DISENT-KWS is a lightweight, real-time dual-gate Keyword Spotting (KWS) and Speaker Verification system. Designed for resource-constrained edge devices, it allows users to register custom wake words and speaker profiles with just 5 audio samples.

The model uses a Disentangled Dual-Head Architecture built on a shared BC-ResNet-2 backbone. An adversarial Gradient Reversal Layer (GRL) and Contrastive Log-ratio Upper Bound (CLUB) mutual information minimizer ensure that phonetic (keyword) and speaker identity representations remain strictly orthogonal. This prevents the system from triggering on the wake word if spoken by an unauthorized user.

Architecture Overview

The system uses a dual-head disentangled architecture built on a shared BC-ResNet-2 encoder (total 1.806M parameters):

1. Shared Encoder (BC-ResNet-2) — Broadcasted residual network, 33.8K params
2. Temporal Block (Mamba SSM / Dilated Conv1D) — O(T) temporal context modeling, 10.3K params
3. Phonetic Head (Causal Conformer) — Extracts keyword-discriminative embeddings zₚₕₙ ∈ ℝ¹⁹², 1,673K params
4. Speaker Head (ECAPA-TDNN Lite) — Extracts speaker-discriminative embeddings zₛₚₖ ∈ ℝ¹⁹², 88.8K params
5. Disentanglement Module (GRL + CLUB) — Adversarial gradient reversal + mutual information minimization forces zₚₕₙ ⟂ zₛₚₖ
6. Dual-Gate Scorer — Weighted cosine similarity (w_kw=0.30, w_spk=0.65) with EMA smoothing and DET-calibrated threshold (τ=0.2222)

---

Datasets Used

Dataset	Usage	Samples	License
Google Speech Commands v2	Keyword spotting pre-training	105K utterances	CC BY 4.0
VoxCeleb1	Speaker verification training	153K utterances (1,251 speakers)	CC BY 4.0
LibriPhrase	Hard-negative triplet pairs (up to 3K triplets generated from metadata)	Apache 2.0
MUSAN	Noise augmentation	109 hrs	CC BY 4.0

---

Final Performance Benchmarks

Evaluated on Google Speech Commands v2 test set (11,005 samples, 35 classes) and VoxCeleb1 (1,251 speakers). Scorer weights calibrated via joint verification grid search over 10×10 weight combinations + DET-driven threshold selection.

Metric	Achieved	Target	Status
Parameters	1.806 M	< 3.0 M	✅
ONNX Model Size	0.60 MB (INT8)	—	✅
CPU Latency	26.43 ms (p95: 28.29 ms)	< 200 ms	✅
Real-Time Factor (xRT)	0.0132	< 0.20	✅
Keyword EER (standalone)	4.69%	low	✅
Speaker EER (standalone)	17.86%	low	✅
Joint EER	23.47%	—	—
Joint AUC	0.8425	—	✅
Optimal Scorer Weights	wₖw=0.30, wₛₚₖ=0.65	—	✅
EER Threshold (τ)	0.2222	—	✅

---

License & Attribution

This project is licensed under the MIT License.

It transfers weights from and builds upon the open-source SpeechBrain repository (ECAPA-TDNN for speaker verification), licensed under Apache 2.0.

Team: Noisy AF

• Sohini Banerjee
• Swarnim Tripathi
  VIT Chennai