vocal-burst-detector-x2

Which file do I want?

production/ — start here. The instrument this repo exists to ship: a 5-member ensemble on the laion/voiceclap-large-v2 encoder, trained on both halves of laion/vocal-bursts-segments, with bs_score.py (an importable scorer), per_class_recall.json (read this before you read any hit rate) and the full confusion matrices. Jump to The production head.

The repo root is a drop-in for vocal_burst_mlp_v2.pt — same frozen 768-d FastScorer features, so it swaps into an existing reward.py without touching the feature path. It is beaten by production/ in every cell and is kept for compatibility, not for accuracy.

voiceclap/ is the five-split-seed research arm behind the encoder choice; its numbers, not its weights, are what you want.

A 17-class vocal-burst head — including a no_burst reject class, which laion/vocal-burst-detector-v2 has but was never trained against verified negatives — on top of that detector's own frozen 768-d feature extractor. It is a drop-in replacement for the shipped vocal_burst_mlp_v2.pt: same embedder, same 768 → 256 → C architecture, new output layer.

Five seeds ship. vocal_burst_mlp_x2_s{0..4}.pt each hold arch, classes and state_dict.

What it is measured to do, and why the measurement is the cross-source one

A within-source split cannot answer "how robust is it": training and test labels both come from gemini-3.8-flash, so the model is graded by the standard it was trained on. The informative measurement trains on one generator and tests on the other. Real speech and DramaBox TTS have nothing acoustic in common.

train → test	balanced acc	burst vs no-burst	neg speech	neg silence	shipped (restricted)	shipped (83-way)
real->real	43.4 % ± 0.4	97.6 %	96.1 %	70.4 %	26.6 %	14.8 %
real->dramabox	34.2 % ± 1.5	92.8 %	95.3 %	83.9 %	25.5 %	14.8 %
dramabox->real	34.3 % ± 1.0	94.7 %	95.8 %	26.4 %	26.6 %	14.8 %
dramabox->dramabox	50.4 % ± 0.3	97.4 %	97.9 %	82.0 %	25.5 %	14.8 %
both->real	38.2 % ± 1.7	96.3 %	96.7 %	28.0 %	26.6 %	14.8 %
both->dramabox	51.2 % ± 1.6	97.2 %	98.2 %	79.7 %	25.5 %	14.8 %

Chance = 5.9 % over 17 classes. Five seeds, grouped splits (real: by speaker; DramaBox: by prompt, so all three seeds of one sentence move together). The test set for a source is fixed per seed and reused by every arm.

neg speech and neg silence are reported separately and never pooled. Silence is a trivially easy negative; in production every decision is speech-vs-burst. Where the two disagree, the speech column is the real one.

Per class

class	accuracy
no_burst	82.4 %
Scream	76.8 %
Panting	72.8 %
Sharp Inhale	60.8 %
Chuckle	60.0 %
Affirmative Grunt	53.6 %
Frustrated Groan	53.6 %
Soft Hum	38.4 %
Breathy Giggle	33.6 %
Yawn	26.4 %
Exhausted Groan	21.6 %
Humming	18.4 %
Wistful Sigh	15.2 %
Heavy Breathing	12.0 %
Exasperated Sigh	11.2 %
Relief Sigh	8.8 %
Deep Breath	3.2 %

Use

import torch, torch.nn as nn
ck = torch.load("vocal_burst_mlp_x2_s0.pt", map_location="cpu", weights_only=False)
a = ck["arch"]
net = nn.Sequential(nn.Linear(a["D"], a["H"]), nn.BatchNorm1d(a["H"]), nn.GELU(),
                    nn.Dropout(a["dropout"]), nn.Linear(a["H"], a["C"]))
net.load_state_dict({k.replace("net.", ""): v for k, v in ck["state_dict"].items()})
# features: FastScorer.emb.encode_waveform on a 16 kHz mono cut, exactly as vocal-burst-detector-v2

The production head — production/

This is the instrument to re-score with. Encoder laion/voiceclap-large-v2 (3584-d, frozen), head 3584 → 256 → 17, five initialisations averaged in probability space, trained on both halves of laion/vocal-bursts-segments. The root of this repo is still the FastScorer drop-in; production/ is the better instrument and the one that should be used for new measurements.

An ensemble across split seeds would be dishonest to score — seed k's member has trained on seed j's test items — so the ensemble uses one grouped split, five initialisations inside it, and is scored on that split's held-out groups, which no member has seen.

held-out set	n	exact acc	family acc
real, all held-out	1388	75.4 %	84.8 %
real, balanced 25/class	425	46.6 %	68.7 %
dramabox, all held-out	600	63.3 %	79.3 %
dramabox, balanced 25/class	425	57.4 %	75.8 %

Chance is 5.9 % over 17 classes. The all held-out rows follow the corpus's own class prior and are the right rows for reading recall; the balanced rows are the ones comparable to the cross-source table above.

⚠ Read the per-class recall before reading any hit rate

A hit-rate table that says "burst class X is produced at rate H" is measuring two things at once: the generator producing the sound, and this detector being able to name it. They multiply. A class this detector recalls at 20 % cannot show a hit rate meaningfully above 20 %, however good the generator is — so without the recall column, a low row is unreadable: you cannot tell a silent generator from a deaf instrument. With it you can: a hit rate near its class's recall means the detector was the limit; far below it means the generator was.

production/per_class_recall.json and .csv ship for exactly this. Classes marked ⚠ have fewer than 30 held-out examples in one of the two sources and their recall is not a solid estimate.

Family recall is beside strict recall on purpose. The failure mode here is granularity, not deafness — Breathy Giggle → Chuckle, Exhausted Groan → Frustrated Groan. A recipe that reliably produces a groan but gets named the wrong groan is useful, and reporting it as a failure without the family column would be wrong.

class	real n	recall	family	dramabox n	recall	family	top confusions (dramabox)
Affirmative Grunt ⚠	30	70.0 %	76.7 %	25	68.0 %	68.0 %	Humming 4, Chuckle 2
Breathy Giggle ⚠	32	46.9 %	96.9 %	28	85.7 %	100.0 %	Chuckle 4
Chuckle	53	64.1 %	84.9 %	40	80.0 %	97.5 %	Breathy Giggle 7, Panting 1
Deep Breath	59	18.6 %	71.2 %	51	68.6 %	86.3 %	Sharp Inhale 9, Exasperated Sigh 2
Exasperated Sigh	53	22.6 %	34.0 %	42	38.1 %	59.5 %	Relief Sigh 8, Deep Breath 7
Exhausted Groan ⚠	31	48.4 %	87.1 %	25	32.0 %	76.0 %	Frustrated Groan 11, Exasperated Sigh 2
Frustrated Groan ⚠	32	59.4 %	81.2 %	26	65.4 %	65.4 %	Yawn 3, Exasperated Sigh 2
Heavy Breathing ⚠	25	12.0 %	64.0 %	25	28.0 %	92.0 %	Panting 9, Deep Breath 7
Humming	36	47.2 %	61.1 %	31	51.6 %	80.7 %	Soft Hum 9, Affirmative Grunt 2
Panting ⚠	34	64.7 %	88.2 %	26	46.2 %	57.7 %	Chuckle 5, Exasperated Sigh 3
Relief Sigh	32	6.2 %	31.2 %	32	46.9 %	75.0 %	Deep Breath 7, Exasperated Sigh 7
Scream ⚠	37	59.5 %	59.5 %	27	96.3 %	96.3 %	Breathy Giggle 1
Sharp Inhale	39	89.7 %	89.7 %	32	53.1 %	75.0 %	Deep Breath 7, Exasperated Sigh 3
Soft Hum ⚠	28	39.3 %	57.1 %	25	28.0 %	60.0 %	Humming 8, Affirmative Grunt 4
Wistful Sigh ⚠	28	14.3 %	39.3 %	25	40.0 %	56.0 %	Exasperated Sigh 3, Exhausted Groan 3
Yawn	37	46.0 %	46.0 %	30	50.0 %	50.0 %	Exhausted Groan 3, Frustrated Groan 3
no_burst	802	98.0 %	98.0 %	110	96.4 %	96.4 %	Sharp Inhale 3, Deep Breath 1

How to call it

import sys; sys.path.insert(0, "<dir containing bs_score.py>")   # ships in production/
from bs_score import ProductionBurstScorer

sc  = ProductionBurstScorer(device="cuda", ckpt_dir="<.../production>")
out = sc.bursts(wavs16)          # list[np.float32 @16 kHz] -> list[list[dict]]

out[i] is the bursts in clip i, in the same shape reward.RewardModel.bursts returns, so gl_run.py / vc_run.py-style code swaps instrument without changing its aggregation:

{"start": 3.12, "end": 3.74, "dur": 0.62,
  "label": "Scream", "prob": 0.83,
  "labels": [["Scream", 0.83], ["Shriek", 0.07], ["Frustrated Groan", 0.03]],
  "family": "scream", "no_burst_prob": 0.02}

• sc.bursts(wavs16, no_burst_gate=0.5) — locate with laion/vocalburst-locator v2 (unchanged constants, so a span is the same span the old stack reports; only the name changes), then name, dropping spans the head rejects.
• sc.classify(cuts16) — name spans you already cut; drops nothing, apply your own gate.
• sc.recall_floor() — the table above as {class: {source: …}}.

python bs_score.py runs a self-test.

A second encoder was tried: laion/voiceclap-large-v2

Single-variable arm — same segments, same grouped splits, same head shape, same five seeds; only the encoder changes. The dimension changes with it, so this head is 768 → 256 → 17 while the VoiceCLAP one is 3584 → 256 → 17, and the VoiceCLAP arm has no shipped-detector baseline and no warm start because that detector cannot consume a 3584-d vector.

train → test	FastScorer 768-d (this model)	VoiceCLAP 3584-d	Δ
real->real	43.4 %	45.6 %	+2.2 pts
real->dramabox	34.2 %	40.6 %	+6.4 pts
dramabox->real	34.3 %	35.7 %	+1.4 pts
dramabox->dramabox	50.4 %	56.0 %	+5.6 pts
both->real	38.2 %	44.9 %	+6.8 pts
both->dramabox	51.2 %	57.6 %	+6.3 pts

Sanity check on the same 120 segments before either was trained on: 1-NN class accuracy 78.3 % (VoiceCLAP) against 74.2 % (FastScorer), chance 25 %. VoiceCLAP throughput was measured at 15.6 segments/s on one GPU (3.1 GPU-hours for the whole corpus).

One conclusion changes with the encoder and it is the most useful thing here: with the 768-d extractor, adding the DramaBox half cost 5.2 points on real speech (both→real 38.2 % against real→real 43.4 %); with VoiceCLAP the same comparison is 44.9 % against 45.6 % — under a point. The harm from mixing the two sources was largely an encoder limitation, not a property of the data. The one place VoiceCLAP is worse: a head trained on DramaBox alone recognises real-speech negatives less well (84.1 % against 95.8 %) — the richer encoder also encodes more of the generator.

Both ship. The root of this repo is the FastScorer head — the drop-in for vocal_burst_mlp_v2.pt, consuming exactly the vectors reward.py already computes. voiceclap/vocal_burst_mlp_vclap_s{0..4}.pt are the VoiceCLAP heads (3584 → 256 → 17); they need laion/voiceclap-large-v2 at inference (~18 GB, ~16 segments/s on one GPU), which is the price of the extra points.

What it cannot do

• It is a classifier, not a locator. It labels a span something else has proposed. The spans it was trained on are gemini-3.8-flash's, which this project measured as better at what and worse at where than the shipped locator.
• Its labels have never been checked by a human. Every training and test label is one model's opinion. A shared prior between that model and the detector whose features this uses would inflate every number above and nothing measured here can see it.
• Half its training data is TTS output. The DramaBox half of laion/vocal-bursts-segments is generated audio whose LTX-2 Community Licence has not been assessed; if that matters for your use, retrain from the real config alone — the real->real and real->dramabox rows above are what that head scores.

Class groups

vocal_burst_groups.json and GROUPS.md carry a 23-group scheme over 117 burst label strings, grouping names that denote the same or a very similar sound (snicker/chuckle, shriek/scream, cough/coughing). Scoring the same predictions at group level raises the mean generation hit rate from 0.302 to 0.537; a random grouping with identical group sizes reaches 0.355, so +0.182 of it is the grouping being right and the rest is arithmetic. Groups were checked with directed lift rather than raw confusion, because two labels account for 29 % of all annotator top-1 calls whatever was requested and merging on raw confusion books a generation failure as a hit.

For training the classifier, keep the fine classes: collapsing them raises raw accuracy only because chance rises with it. Group at evaluation time — that can be done at any point, the reverse cannot.