Gemma 4 spoken language detector (work in progress)
Spoken language identification from one second of audio, built from the
first five blocks of the Gemma 4 E2B audio tower with a small linear head
on mean/std-pooled features. The tower slice is frozen Gemma weights
(127M parameters); each head is about 100k parameters. Everything is run
with Elixir/Nx from
olafura/gemma-4-mic-transcribe
(language_id detect | serve | validate), on CPU through Torchx or on an
Nvidia GPU through EXLA.
This is a WIP snapshot, not a release: the weak languages below are weak because of training data, not by design.
Files
| folder | languages | head fitted on |
|---|---|---|
sent49/ |
49 | Common Voice sentence onsets, up to 1500 per language, 47.6k clips |
mixed34/ |
34 | 18 sentence-onset languages plus 16 single-word languages (incl. zh) |
Each folder is one language_id artifact: parameters.safetensors (tower
slice plus head, 254 MB) and manifest.etf (languages, depth, window,
pooling, parameter paths). Load it with
./language_id serve --artifact sent49 --port 7861
curl -s -X POST --data-binary @clip.wav 'localhost:7861/detect?languages=en,de,sv-SE'
Accuracy
Top-1 / top-3 on held-out speakers, one second of audio, all languages open unless stated:
| head | test set | top-1 | top-3 |
|---|---|---|---|
| sent49 | Common Voice sentence onsets, 49 languages, 1436 clips | 48.7% | 66.4% |
| sent49 | same, 18 best-covered languages | 70.0% | 87.4% |
| sent49 | same, 18 languages told as candidates | 76.1% | 90.9% |
| mixed34 | Common Voice sentence onsets, its 18 sentence languages | 75.2% | 89.3% |
| sent49 | fixie-ai/language_detection-audio, 21 candidates, first 1 s | 46.0% | 64.3% |
| sent49 | same, 1 s windows summed over the whole clip | 57.7% | 74.7% |
On the last set the 32B Ultravox (v0.6, Qwen3-32B) gets 45.0% from the same first second and 89.3% from whole clips. Model time per one-second clip: 93 ms on an A100, 143 ms on a Strix Halo CPU.
Languages with fewer than 1000 training onsets, where the head mostly answers a neighbour: te 8, oc 34, ko 47, ast 49, ml 158, sl 174, mk 211, sr 235, el 240, ha 241, fi 260, mn 272, mr 277, vi 288, et 395, sk 408, da 436, hi 587, bg 607, ro 643, ur 671, lt 907.
Training data
Common Voice 17 (fixie-ai/common_voice_17_0), train split, speakers disjoint from the test shards. Base model: google/gemma-4-E2B-it, subject to the Gemma terms of use; the head and code are Apache-2.0.