laion/vocal-burst-classifier-gemini

Vocal-burst classifier head, retrained on Gemini-annotated segments

An 11-class classifier head for non-verbal vocal bursts — laughs, sighs, gasps, screams — that replaces the head this project had been using. 56.1 % ± 3.4 accuracy against 39.0 % for the previous head on the same eleven classes, with 9.1 % chance.

Why it exists

The previous detector was the measuring instrument for every vocal-burst experiment in this project, and an audit found it could barely name what it heard: over 60 clips it used 8 of its 83 labels, emitted Shriek zero times, put 70 % of its detections on two labels, and had the requested burst in its top-3 on 3 of 60 clips. Every burst adapter here was trained on its labels. This head is the first step out of that loop — trained on the same kind of audio, re-annotated blind by Gemini 3.8 Flash.

Results

Five seeds, speaker-disjoint splits: whole speakers go to train, val or test and are never split across them. Several segments are cut from one utterance and many utterances come from one voice profile, so a random split would put the same breath on both sides and report memorisation.

model	accuracy over the 11 classes
this head	56.1 % ± 3.4
previous head, restricted to the same 11 classes	39.0 %
previous head, its full 83 classes	18.9 %
chance	9.1 %

Both baselines are given on purpose. The restricted number is the fair comparison, since this head cannot emit the other 72 classes; the unrestricted one is what the previous head actually did in the pipeline. Quoting only one would flatter one side.

class	training segments	accuracy
Chuckle	243	89 %
Exhausted Groan	138	76 %
Humming	176	72 %
Scream	155	71 %
Panting	263	67 %
Sharp Inhale	243	63 %
Relief Sigh	104	53 %
Exasperated Sigh	393	41 %
Yawn	174	33 %
Deep Breath	428	32 %
Heavy Breathing	134	20 %

Scream at 71 % is the one that matters here: screams are exactly what the previous detector could never hear, which is why its scream adapters could never be evaluated.

checkpoint	test accuracy	test segments
vocal_burst_mlp_gem_s0.pt	58.2 %	165
vocal_burst_mlp_gem_s1.pt	55.8 %	165
vocal_burst_mlp_gem_s2.pt	58.8 %	165
vocal_burst_mlp_gem_s3.pt	57.6 %	165
vocal_burst_mlp_gem_s4.pt	50.3 %	165

All five are shipped rather than only the best. Picking the best seed by its own test score reports a number the model cannot reproduce; the honest headline is the mean, and the spread is the ± above. Averaging the five heads' logits is a free ensemble.

How much to believe this

The test labels come from Gemini, so this head is judged by the standard it was trained on and the previous head by one it never agreed with. Part of the 17-point gap is that home advantage, and how much has not been measured — that needs a human listening pass, which has not happened yet. What the number does establish is that the class is learnable from these segments at all: at 11 classes and 9.1 % chance, 56 % is not noise.

The two annotators also disagree about granularity, not vocabulary: on the source corpus they agree on the burst family 64.8 % of the time but on the exact label only 19.0 %.

Architecture and use

The 768-d embedder is frozen and unchanged — the same one the previous head used. Only the MLP head is trained, keeping the shipped shape (768 → 256, BatchNorm, GELU, Dropout 0.3 → 11) and warm-starting its trunk from the previous checkpoint. The output layer is new because the label set is new. This is a tuning of the shipped classifier, not a different model in the same slot.

• input: 16 kHz mono waveform of one already-located burst segment (median 0.76 s)
• embedding: 768-d, frozen
• output: 11 logits; the class order is in classes inside each checkpoint

import torch, torch.nn as nn
ck = torch.load("vocal_burst_mlp_gem_s0.pt", map_location="cpu")
head = nn.Sequential(nn.Linear(768, 256), nn.BatchNorm1d(256), nn.GELU(),
                     nn.Dropout(0.3), nn.Linear(256, len(ck["classes"])))
head.load_state_dict(ck["state_dict"]); head.eval()

emb  = your_frozen_768d_embedder(waveform_16k)      # (B, 768)
pred = [ck["classes"][i] for i in head(emb).argmax(-1)]

Feeding 48 kHz audio to a 16 kHz embedder silently halves every pitch — resample first.

Training data

laion/vocal-bursts-gemini-segments — the 2,451 segments with no overlapping speech in the eleven classes that hold at least 100 examples, from 842 distinct speakers.

Each epoch draws exactly K indices per class with replacement, so the loss sees a uniform class prior while every row stays available; truncating every class to the smallest (104) would have discarded 58 % of the data. The test set is balanced by construction, so its accuracy is directly comparable to 1/11.

bc_feat.py and bc_train.py are included — the feature extraction and the training run exactly as used, not a cleaned-up retelling.

Limitations

• 11 classes, not 83. The others had fewer than 100 segments; several have fewer than 20 in the entire corpus. This head is silent about them — it does not reject them.
• Segments, not streams. It classifies a burst that has already been located. It is not a detector and not a locator: the span boundaries in its training data were never validated, and the dataset card says so explicitly.
• Heavy Breathing (20 %), Deep Breath (32 %) and Yawn (33 %) are at or below usable. The breath family is where both the annotator and this head are weakest.

---

Correction (2026-09-04)

An earlier version of this card said the shipped vocal-burst-detector-v2 "used 8 of its 83 labels" and "emitted Shriek zero times". That was measured on a 60-clip audit and does not generalise. Run over both full corpora (72,500 DramaBox clips and 3,598 real utterances), the same detector emits 41 distinct labels on the DramaBox half and 36 on the real half, and it does emit Shriek — on 50 DramaBox clips and 49 real ones.

What the 60-clip audit actually saw is how concentrated the distribution is: Contented Sigh alone accounts for 16,694 of 72,500 DramaBox clips, Surprised Gasp 7,451, Wistful Sigh 5,437. The correct statement is that the detector's effective vocabulary is small, not that its emitted vocabulary is 8. The conclusions drawn elsewhere in this card — that the detector is the bottleneck, and that its failure is granularity rather than deafness — are unaffected and are independently supported by the cross-source results in §62.