vocal-burst-detector-x3

Which file do I want?

production/ — start here. The laion/voiceclap-large-v2 (3584-d, frozen) head, 5 initialisations averaged in probability space, 19 classes = 18 bursts + no_burst, with bs_score.py (importable scorer, unchanged API), per_class_recall.json (read it before you read any hit rate) and the full confusion matrices. commercial/ is the same recipe on laion/voiceclap-commercial (768-d): cheaper, and — see below — the more selective rejecter. ablations/ holds the variants that explain every number here. training_extras/ carries the provenance of the new negatives.

laion/vocal-burst-detector-x2 is untouched; this repository is its successor for re-scoring the laion-tts-annotated-v1 corpus.

What changed against x2, and why — three findings

x2 was used to re-classify the 63.8 M burst spans that laion/vocalburst-locator v2 + the old vocal-burst-detector-v2 had stored in laion-tts-annotated-v1. The result was that 70 % of all spans came back as Sharp Inhale (Mediathek: 81 %), and that the two commonest old labels — Ahem and Low Mumble, ~26 % of the old events each — were not in x2's 16-class vocabulary. The obvious reading was "x2 cannot name the two commonest sounds; add them". Three measurements turned that reading around (all in training_extras/ and the report this card summarises):

1. The old detector invented its two commonest labels. In the 3,598 real utterances that carry both the old spans and a blind whole-clip gemini-3.8-flash annotation, a Gemini event overlaps an old Ahem span in 16.5 % of cases and an old Low Mumble span in 7.3 % — against 86.5 % for old Chuckle spans and 91.5 % for old Sharp Inhale spans. A level dip (≤ −10 dB) does not change this (15.2 % / 5.0 %). Where an event does overlap, it is almost never a throat-clear or a mumble (6 and 1 of ~1,250). Those spans are locator false alarms mid-speech, given the old detector's two fallback names.
2. x2 never rejects them. On the old spans in the corpus, x2's P(no_burst) is 0.000 at the median and ≥ 0.5 on 0.03–0.09 % of them, whatever the level contrast. Its no_burst training rows were ≥ 0.5 s windows of speech or silence, ≥ 0.5 s clear of every annotation; a 0.2–0.5 s cut inside running speech is something it never saw as "nothing", so it picks the nearest burst class, and for a short breathy speech fragment that is Sharp Inhale. A calibration problem of the reject output, not a vocabulary problem.
3. A real throat-clear is not what x2 calls Sharp Inhale either. On Gemini's own Ahem segments, x2 says Chuckle (real, 50 %) or Humming / Affirmative Grunt / Chuckle (DramaBox); on Low Mumble it says Humming. So the two classes are missing from the vocabulary — but adding them alone would have forced the same non-events into another class.

x3 therefore does two things: it adds Ahem and Low Mumble (Gemini labels Clears Throat and Throat-clearing folded into Ahem, Whispered Mumble into Low Mumble; Cough deliberately not folded in — a cough is a different sound), and it adds a third no_burst sub-type, fp_span: old-detector spans labelled Ahem / Low Mumble that no Gemini event overlaps — 1,115 from the real utterances and 1,200 mined the same way on DramaBox (locator v2 + old detector on 35,224 clips, so that fp_span is not learnable as "is real audio"). The negative mix per epoch is 0.55 speech / 0.20 silence / 0.25 fp_span (x2: 0.75 / 0.25). Everything else is x2's recipe verbatim: one grouped split (real by speaker, DramaBox by prompt), split seed 0, five initialisations, 120 epochs, 3584 → 256 → 19 (768 → 256 → 19), dropout 0.3, 25 held-out per class.

The two new classes are 96–97 % DramaBox material. Gemini's real-audio half has 44 Ahem and 14 Low Mumble segments (x2's rule of ≥ 100 per half would have admitted neither); they were admitted on the pooled count (1,099 / 418). Their real-audio recall below is measured on those few rows and flagged ⚠ accordingly. Re-annotating old Ahem / Low Mumble spans to get more real examples was costed and dropped: finding 1 puts the expected yield at ~1 %.

The number this repository exists for: rejection of the false-alarm spans

no_burst recall on held-out rows by sub-type (fp_span rows are speaker-/prompt-disjoint from training; x2 never trained on any fp_span row, so all 2,315 of them are fair for x2 — it rejects 0.2 %):

sub-type	n	x2 (large-v2)	x3 production/	x2 (commercial)	x3 commercial/
real / fp_span	60	0 %	97 %	2 %	92 %
dramabox / fp_span	26	0 %	96 %	0 %	92 %
real / speech	777	99 %	99 %	97 %	96 %
real / silence	25	56 %	44 %	36 %	36 %
dramabox / speech	89	100 %	99 %	99 %	99 %
dramabox / silence	49	92 %	88 %	84 %	82 %

On the 1,967 stored spans of 3,200 validation clips (emolia + podcast, never seen by either head; */validation_spans_x2_vs_x3.md), gate P(no_burst) ≥ 0.5:

old label	n	x2 rejects	x3 production/ rejects	x3 commercial/ rejects
Low Mumble	1002	0.0 %	97.2 %	89.4 %
Ahem	701	0.0 %	97.4 %	76.9 %
Surprised Gasp	54	0.0 %	74.1 %	16.7 %
Wistful Sigh	31	0.0 %	74.2 %	29.0 %
Contented Sigh	24	0.0 %	50.0 %	4.2 %
Exhausted Groan	28	0.0 %	50.0 %	0.0 %
Childlike Giggle	44	0.0 %	22.7 %	9.1 %
Chuckle	38	0.0 %	18.4 %	0.0 %
Breathy Giggle	35	0.0 %	37.1 %	0.0 %
all	1967	0.0 %	90.7 %	74.2 %

Sharp Inhale as top-1 goes from 70.2 % of spans (x2) to 1.2 % (large-v2) / 3.4 % (commercial). The rejection is flat across level-dip bands (≤ −20 dB: 94 %; > 0 dB: 91 %), i.e. it is not a level detector.

Is it rejecting the right spans? Verified against unverified

All 5,143 old spans of the 3,598 Gemini utterances, split by whether a Gemini event overlaps the span (*/rejection_verified_vs_unverified.md). "Unseen" = the utterance contributed no fp_span training row.

spans	n	x2	x3 production/	x3 commercial/
unverified (no Gemini event), old Ahem/Low Mumble	1115	0.1 %	98.7 % (training rows)	88.0 %
unverified, other old labels, unseen utterances	482	0.0 %	40.7 %	6.2 %
verified (a Gemini event overlaps), unseen utterances	2996	0.0 %	13.7 %	2.6 %
verified Chuckle	540	0.0 %	20.6 %	3.1 %
verified Scream	164	0.0 %	6.7 %	0.6 %
verified Sharp Inhale	105	0.0 %	21.0 %	6.7 %

So the two heads trade differently: production/ rejects more, including 14 % of verified bursts at gate 0.5; commercial/ rejects almost only the Ahem/Low Mumble-type false alarms and keeps 97 % of verified bursts. P(no_burst) is delivered ungated, and the gate moves the trade-off (ablations/gate_sweep.json):

gate on P(no_burst)	production: verified rejected	production: old Ahem/LM rejected (validation)	commercial: verified	commercial: old Ahem/LM
0.5	13.7 %	97.3 %	2.6 %	84.3 %
0.7	10.0 %	95.1 %	1.6 %	69.3 %
0.9	5.9 %	91.1 %	0.6 %	40.8 %
0.95	4.7 %	86.6 %	0.4 %	26.4 %

A reasonable default for production/ on corpus spans is a gate of 0.9; for commercial/, 0.5.

What the new negatives cost on burst classification

The old classes are the same 16 and go through the same admission rule; what changes their numbers is (a) two more classes to confuse with and (b) the fp_span negatives. Both were isolated (ablations/, large-v2, same split seed, x2 re-run reproduces x2's release exactly at 0.4659):

variant (large-v2)	real, balanced, same rows as x2	real, all held-out	dramabox, all held-out
x2 (17 classes, no fp_span)	reference	reference	reference
17 classes + fp_span	−7.3 pts (init 5–9: −5.2)	+0.4	+0.3
19 classes, no fp_span	−1.7	−0.1	−1.1
x3 = 19 classes + fp_span	−6.3	−0.1	+2.9
x3 with fp_span at 10 % of the negative draws	−8.5	−0.2	+2.5
initialisation noise (17 classes, init 5–9 vs 0–4)	−0.9	−0.2	+1.8

The balanced real-audio loss comes from the fp_span negatives, is consistent across initialisations, does not shrink when the mix share is cut, and lands on Frustrated Groan, Breathy Giggle, Exhausted Groan, Affirmative Grunt and Sharp Inhale — which move into their neighbour class (groan ↔ groan, giggle → chuckle), not into no_burst or the new classes. On the prior-weighted held-out sets (the rows that look like the corpus) nothing is lost. Across five split seeds (cross-source table below) the 17-class mean on real audio is unchanged (0.454 vs 0.449) and −2.7 pts on DramaBox, so the single-split −6 is partly that split.

If your use is grading generated bursts by class rather than cleaning corpus spans, the variant without fp_span negatives ships as ablations/large-v2_19cls_no_fp_span/ (same 19 classes).

Per class, x2 → x3

large-v2 (production/)

Strict recall on x3's held-out rows (n, x3, family), and the x2 → x3 change on the rows held out by both splits (n_c, x2_c → x3_c; neither head trained on them). ⚠ = fewer than 30 held-out rows in one source.

class	real n	x3	family	n_c	x2_c → x3_c	dbox n	x3	family	n_c	x2_c → x3_c
Affirmative Grunt ⚠	29	66 %	69 %	25	72 % → 60 %	26	73 %	73 %	17	53 % → 59 %
Ahem ⚠ new	26	23 %	23 %	—	—	27	63 %	63 %	—	—
Breathy Giggle	34	29 %	79 %	24	58 % → 25 %	30	63 %	93 %	18	89 % → 67 %
Chuckle	53	77 %	85 %	30	73 % → 77 %	46	65 %	80 %	21	81 % → 81 %
Deep Breath	55	33 %	65 %	42	29 % → 33 %	58	71 %	91 %	24	67 % → 71 %
Exasperated Sigh	69	20 %	32 %	35	17 % → 14 %	49	43 %	59 %	20	60 % → 50 %
Exhausted Groan ⚠	33	52 %	73 %	17	76 % → 53 %	26	31 %	69 %	20	30 % → 30 %
Frustrated Groan ⚠	33	30 %	64 %	25	68 % → 32 %	26	62 %	73 %	19	74 % → 68 %
Heavy Breathing ⚠	34	24 %	71 %	24	33 % → 25 %	25	44 %	88 %	18	22 % → 33 %
Humming	37	51 %	76 %	31	58 % → 48 %	39	59 %	85 %	23	61 % → 61 %
Low Mumble ⚠ new	14	0 %	21 %	—	—	25	28 %	68 %	—	—
Panting ⚠	39	72 %	87 %	31	65 % → 68 %	25	60 %	64 %	10	40 % → 30 %
Relief Sigh	39	8 %	46 %	30	10 % → 3 %	37	43 %	76 %	27	48 % → 44 %
Scream ⚠	37	73 %	73 %	28	68 % → 71 %	27	93 %	93 %	24	96 % → 92 %
Sharp Inhale	52	73 %	81 %	37	89 % → 73 %	41	66 %	83 %	20	60 % → 65 %
Soft Hum ⚠	25	32 %	60 %	22	41 % → 27 %	27	22 %	59 %	24	33 % → 21 %
Wistful Sigh ⚠	26	19 %	35 %	20	15 % → 15 %	27	33 %	44 %	25	36 % → 36 %
Yawn	45	42 %	42 %	34	44 % → 41 %	39	64 %	64 %	22	50 % → 64 %
no_burst	862	97 %	97 %	828	93 % → 97 %	164	95 %	95 %	70	70 % → 94 %

held-out set (large-v2)	n	x3 exact	x3 family	common rows	x2 → x3 on them
real, all held-out	1542	73.2 %	81.6 %	1283	77.9 % → 77.8 %
real, balanced 25/class	464	44.4 %	63.4 %	317	51.4 % → 45.1 %
dramabox, all held-out	764	64.3 %	79.1 %	402	59.0 % → 61.9 %
dramabox, balanced 25/class	475	56.8 %	75.6 %	261	54.8 % → 55.2 %

no_burst sub-type (large-v2, held-out)	n	x2 recall	x2 mean P(no_burst)	x3 recall	x3 mean P(no_burst)
real/speech	777	99 %	0.984	99 %	0.974
real/silence	25	56 %	0.433	44 %	0.328
real/fp_span	60	0 %	0.002	97 %	0.967
dramabox/speech	89	100 %	0.990	99 %	0.985
dramabox/silence	49	92 %	0.868	88 %	0.830
dramabox/fp_span	26	0 %	0.001	96 %	0.948

commercial (commercial/)

Strict recall on x3's held-out rows (n, x3, family), and the x2 → x3 change on the rows held out by both splits (n_c, x2_c → x3_c; neither head trained on them). ⚠ = fewer than 30 held-out rows in one source.

class	real n	x3	family	n_c	x2_c → x3_c	dbox n	x3	family	n_c	x2_c → x3_c
Affirmative Grunt ⚠	29	45 %	45 %	25	52 % → 40 %	26	69 %	73 %	17	65 % → 53 %
Ahem ⚠ new	26	19 %	19 %	—	—	27	70 %	70 %	—	—
Breathy Giggle	34	35 %	71 %	24	33 % → 38 %	30	73 %	90 %	18	89 % → 89 %
Chuckle	53	62 %	72 %	30	67 % → 63 %	46	57 %	70 %	21	76 % → 71 %
Deep Breath	55	5 %	55 %	42	7 % → 7 %	58	53 %	79 %	24	50 % → 50 %
Exasperated Sigh	69	9 %	16 %	35	9 % → 11 %	49	29 %	45 %	20	50 % → 35 %
Exhausted Groan ⚠	33	15 %	58 %	17	35 % → 18 %	26	19 %	46 %	20	15 % → 15 %
Frustrated Groan ⚠	33	42 %	58 %	25	60 % → 44 %	26	62 %	69 %	19	68 % → 68 %
Heavy Breathing ⚠	34	12 %	71 %	24	12 % → 12 %	25	68 %	92 %	18	72 % → 67 %
Humming	37	14 %	46 %	31	19 % → 13 %	39	41 %	67 %	23	43 % → 43 %
Low Mumble ⚠ new	14	14 %	14 %	—	—	25	48 %	60 %	—	—
Panting ⚠	39	72 %	82 %	31	77 % → 71 %	25	64 %	72 %	10	50 % → 50 %
Relief Sigh	39	10 %	41 %	30	7 % → 7 %	37	27 %	62 %	27	30 % → 26 %
Scream ⚠	37	84 %	84 %	28	89 % → 86 %	27	81 %	81 %	24	79 % → 83 %
Sharp Inhale	52	54 %	60 %	37	65 % → 54 %	41	44 %	59 %	20	55 % → 45 %
Soft Hum ⚠	25	44 %	44 %	22	36 % → 36 %	27	19 %	52 %	24	42 % → 12 %
Wistful Sigh ⚠	26	12 %	23 %	20	10 % → 10 %	27	44 %	48 %	25	56 % → 48 %
Yawn	45	20 %	20 %	34	32 % → 24 %	39	44 %	44 %	22	36 % → 36 %
no_burst	862	94 %	94 %	828	91 % → 94 %	164	93 %	93 %	70	67 % → 91 %

held-out set (commercial)	n	x3 exact	x3 family	common rows	x2 → x3 on them
real, all held-out	1542	66.7 %	74.6 %	1283	71.9 % → 72.8 %
real, balanced 25/class	464	34.5 %	52.4 %	317	39.4 % → 36.3 %
dramabox, all held-out	764	58.6 %	70.9 %	402	56.2 % → 56.0 %
dramabox, balanced 25/class	475	52.4 %	66.5 %	261	55.2 % → 51.7 %

no_burst sub-type (commercial, held-out)	n	x2 recall	x2 mean P(no_burst)	x3 recall	x3 mean P(no_burst)
real/speech	777	97 %	0.933	96 %	0.914
real/silence	25	36 %	0.194	36 %	0.180
real/fp_span	60	2 %	0.032	92 %	0.792
dramabox/speech	89	99 %	0.955	99 %	0.935
dramabox/silence	49	84 %	0.411	82 %	0.352
dramabox/fp_span	26	0 %	0.014	92 %	0.772

Cross-source 2×2 (five split seeds, balanced 25/class, one head per seed and arm)

Trained on the left, tested on the right; chance = 5.3 % over 19 classes (x2: 5.9 % over 17). The "17-class mean" column averages x3's per-class recall over the 16 old classes + no_burst so it is comparable to x2's number in the same row.

cell	x3 large-v2 (19)	x3, 17-class mean	x2 large-v2 (17)	Ahem	Low Mumble	x3 commercial (19)	x3, 17-class mean	x2 commercial (17)	Ahem	Low Mumble
real → real	0.422 ± 0.011	0.451	0.456 ± 0.018	0.15	0.00	0.373 ± 0.016	0.402	0.434 ± 0.004	0.08	0.00
real → dramabox	0.400 ± 0.024	0.420	0.406 ± 0.019	0.47	0.00	0.315 ± 0.034	0.351	0.342 ± 0.017	0.02	0.00
dramabox → real	0.342 ± 0.013	0.353	0.357 ± 0.015	0.31	0.06	0.320 ± 0.021	0.328	0.344 ± 0.010	0.32	0.10
dramabox → dramabox	0.545 ± 0.014	0.558	0.560 ± 0.020	0.61	0.26	0.514 ± 0.012	0.511	0.504 ± 0.003	0.67	0.41
both → real	0.430 ± 0.007	0.454	0.449 ± 0.023	0.25	0.04	0.344 ± 0.013	0.362	0.383 ± 0.014	0.19	0.09
both → dramabox	0.536 ± 0.007	0.549	0.576 ± 0.012	0.62	0.24	0.517 ± 0.011	0.512	0.512 ± 0.016	0.72	0.38

The real → arms have 19 real Ahem and no real Low Mumble training rows (all 14 are held out), so their Ahem/Low Mumble cells measure DramaBox-only or near-zero training. Ahem transfers from DramaBox to real speech at 0.25–0.31 (25 real rows per seed; chance 0.05). Low Mumble does not (≤ 0.10) and should be read as a DramaBox-only class until real examples exist.

How to call it

import sys; sys.path.insert(0, "<dir containing bs_score.py>")   # ships in production/
from bs_score import ProductionBurstScorer

sc  = ProductionBurstScorer(device="cuda", ckpt_dir="<.../production>")
out = sc.classify(cuts16)                    # spans you already cut: list[np.float32 @16 kHz]
# each: {"label", "prob", "labels": top-3, "family", "no_burst_prob"} -- apply your own gate
out = sc.bursts(wavs16, no_burst_gate=0.9)   # locate with laion/vocalburst-locator v2, then name

Same API and output shape as x2; the checkpoint carries classes (19, no_burst last), family, label_merge, extra_classes and neg_mix. commercial/ uses bs_vcsmall.py for the encoder and the same checkpoint format. Families: Ahem → throat, Low Mumble → hum.

What it cannot do

• Its labels have never been checked by a human. Every training and test label is one model's opinion (gemini-3.8-flash, blind, whole utterances). The fp_span negatives are defined as "Gemini heard nothing there"; a quiet real breath that Gemini did not annotate is, by that standard, a non-event, and production/ will reject some real breaths (14 % of verified spans at gate 0.5, 6 % at 0.9).
• Low Mumble is a DramaBox class. 14 real examples in total, real recall 0.00–0.14.
• It is a classifier, not a locator. It labels a span something else proposed. On the 3,598 annotated utterances the old locator finds 42 of 44 Gemini Ahem events and 13 of 14 Low Mumble events — but only 23 % of Gemini's Sharp Inhale events and 47 % of Deep Breath.
• Half its training data is TTS output (DramaBox, LTX-2 Community Licence not assessed), and the two new classes almost entirely so.

Files

• production/ — large-v2 head: vocal_burst_mlp_prod_s{0..4}.pt, bs_score.py, bs_vclap.py, bs_det.py, prod_report.json, per_class_recall.{json,csv}, cross_source_2x2.json, validation_spans_x2_vs_x3.{md,json}, rejection_verified_vs_unverified.{md,json}.
• commercial/ — the same for the 768-d head, bs_vcsmall.py.
• ablations/ — large-v2_19cls_no_fp_span/ (heads + reports), large-v2_17cls_with_fp_span/, large-v2_17cls_no_fp_span_x2_rerun/, *_init5-9/, large-v2_19cls_fp_mix_0.10/ (reports), gate_sweep.json.
• training_extras/ — fp_span_negatives.jsonl (2,315 rows: parent clip, span, old label, level dip, why), old_spans_vs_gemini_3598_utterances.json (finding 1), x2_on_gemini_ahem_lowmumble_segments.json (finding 3).
• GROUPS.md, vocal_burst_groups.json — the 23-group scheme, unchanged from x2.