Vocal Burst Classifier V2 — laion/vocal-burst-detector-v2

A single-label classifier over an 83-class vocal-burst taxonomy (82 non-speech human vocalizations + no_burst, index 82). It is a small MLP head on top of frozen VoiceCLAP-commercial audio embeddings.

• Input: any mono waveform (resampled to 16 kHz internally) → VoiceCLAP encode_waveform → 768-d embedding.
• Head: Linear(768,256) → BatchNorm → GELU → Dropout(0.3) → Linear(256,83) (~219k params).
• Output: argmax class + top-k softmax probabilities.

Hear it in a pipeline

25-clip demo grid — locator -> this classifier -> caption ↗

25 multilingual clips in which laion/vocalburst-locator (model_v2.pt, post-processing 0.50 / 0.10 / 0.10) finds each burst and this classifier names it, with the label written inline into a procedural voice-acting caption at the moment the burst happens. Every card has an audio player and lists every located span — kept or dropped — with the reason, so the no_burst veto (P(no_burst) >= 0.5) is auditable next to the audio rather than invisible: on that page 24 of 25 located spans were named and 1 was vetoed.

Results

Metric is argmax accuracy on the official single-label validation set (1940 clips: 1170 pure single-burst + 770 composite), from laion/vocalburst-classification.

model	val acc	pure (1170)	composite (770)	note
V2 mixed emb-MLP (shipped)	58.1%	70.4%	39.4%	existing + Gemini set, 768-d embedding
prior study: mixed emb-MLP	58.61%	72.05%	38.18%	same recipe, different seed
prior release: existing-only emb-MLP	57.89%	70.60%	38.57%	trained on prior data only
Gemini-set-only emb-MLP	19.54%	26.07%	9.61%	new data alone underperforms
baseline shipped multi-label model	0.93%	0.77%	1.17%	multi-label head, not argmax-comparable

On a separate Gemini-audio holdout (the only audio-bearing split, since the official val ships embeddings only), the 768-d VoiceCLAP embedding (37.84%) matched the 1536-d full-last-layer mean+max pooling (36.77%) — the heavier feature did not help, so the shipped model uses the compact 768-d embedding.

How V2 improves on V1: V2 mixes the prior release training data with a new Gemini-annotated set (below), lifting held-out accuracy from 57.89% → 58.1%, with the largest gains on composite clips (38.57% → 39.4%). Adding the new data alone is not enough (19.5%); the mix is what helps.

Training data

Union of two sources (all as precomputed VoiceCLAP-commercial embeddings):

1. Prior release train split from laion/vocalburst-classification (70,176 clips), with the hand classes folded to no_burst.
2. V2 Gemini set (25,245 clips):
   • Gemini-3.6-Flash blind reannotation of laion/vocal-bursts-clean (17,981). The original vocal-bursts-clean labels are very noisy — Gemini-3.6 agreed with them only 5.8% of the time.
   • Gemini-Flash-2.5-validated laion/improved_synthetic_vocal_burts (7,264; ≥250 ms, samples whose intended class agrees with the Flash-2.5 annotation).

Full reproducible embeddings + audio: laion/vocal-burst-classification-v2.

The prior Improved Synthetic Vocal Bursts source contained hand/body impact classes (Finger Snaps, Hand Scratching Head, Hand Slaps, Slap Face). These are not real vocal bursts and were removed — folded into no_burst (index 82) — during feature extraction. They remain in the class list at indices 78–81 for backward compatibility but receive zero training samples.

Inference

from inference import VocalBurstClassifier
clf = VocalBurstClassifier()                 # loads VoiceCLAP-commercial + the V2 head
print(clf.predict("my_clip.wav", topk=5))
# {'prediction': 'Chuckle', 'confidence': 0.32,
#  'topk': [{'class': 'Chuckle', 'prob': 0.32}, {'class': 'Childlike Giggle', 'prob': 0.26}, ...]}

Tested example (a real vocal-bursts-clean clip, Gemini-3.6 ground-truth Contented Sigh):

$ python inference.py clip.mp3 --topk 5
  -> Contented Sigh  (40.1%)
      40.11%  Contented Sigh
      25.43%  Wistful Sigh
      10.24%  Exhausted Groan
       3.68%  Exasperated Sigh
       2.88%  Resonant Hum

Note the full top-5 stays within the sigh / breath cluster — a good illustration of why the demo surfaces top-5 probabilities rather than only the argmax.

Reproduce training

python train.py            # pulls embeddings from the dataset repo, trains, evaluates, saves the checkpoint

Files

• vocal_burst_mlp_v2.pt — shipped checkpoint (state_dict + arch + classes).
• classes.json — 83 class names (index 82 = no_burst).
• class_descriptions.json — one-line description of each class.
• inference.py, train.py.

Full class list

idx	class	group	description
0	Breathy Giggle	laughter	Soft airy laugh.
1	Cackle	laughter	Harsh sharp laugh sometimes unsettling.
2	Childlike Giggle	laughter	Light playful laugh.
3	Chuckle	laughter	Soft suppressed laugh showing mild amusement.
4	Guffaw	laughter	Loud unrestrained laugh of strong amusement.
5	Nervous Giggle	laughter	High-pitched laugh indicating anxiety.
6	Snicker	laughter	Stifled laugh often conveying mockery or sarcasm.
7	Snorting Giggle	laughter	Brief nasal snort combined with laughter.
8	Convulsive Sob	crying_and_distress	Intense crying with heaving breaths.
9	Mournful Wail	crying_and_distress	Long high cry of grief.
10	Quiet Sob	crying_and_distress	Soft crying with short broken breaths.
11	Sobs	crying_and_distress	Audible heavier crying with convulsive gasps.
12	Trembling Whimper	crying_and_distress	Faint cry indicating fear or pain.
13	Deep Breath	breathing	Large inhalation to calm or refocus.
14	Deep Breathing	breathing	Slow deliberate inhalations and exhalations for relaxation.
15	Fast Breathing	breathing	Rapid inhalation and exhalation from excitement or stress.
16	Heavy Breathing	breathing	Deep labored breaths of fatigue or tension.
17	Normal Breathing	breathing	Regular rhythmic breaths at rest.
18	Panting	breathing	Rapid breathing due to exertion or excitement.
19	Slow Breathing	breathing	Paced controlled breaths for calm.
20	Contented Sigh	sighs	Deep exhale expressing satisfaction.
21	Exasperated Sigh	sighs	Audible breath of frustration.
22	Relief Sigh	sighs	Release of tension after stress.
23	Wistful Sigh	sighs	Soft exhale tinged with longing.
24	Fearful Gasp	gasps_and_inhales	Sudden inhale showing shock or anxiety.
25	Sharp Inhale	gasps_and_inhales	Quick breath of sudden realization.
26	Surprised Gasp	gasps_and_inhales	Quick intake of breath in astonishment.
27	Exhausted Groan	groans_and_moans	Weary sound indicating fatigue.
28	Frustrated Groan	groans_and_moans	Deep sound of annoyance.
29	Pain Moan	groans_and_moans	Low vocalization signaling discomfort.
30	Pleasure Moan	groans_and_moans	Soft prolonged sound of enjoyment.
31	Affirmative Grunt	grunts	Short sound indicating agreement.
32	Displeased Grunt	grunts	Harsh vocalization of dissatisfaction.
33	Effort Grunt	grunts	Short forceful sound during physical strain.
34	Ahem	throat_and_vocal_sounds	Clearing throat to gain attention or show discomfort.
35	Clears Throat	throat_and_vocal_sounds	Brief intentional effort to remove phlegm or gain attention.
36	Growl	throat_and_vocal_sounds	Low guttural rumble conveying anger.
37	Gurgling	throat_and_vocal_sounds	Bubbling sound in the throat.
38	Hiss	throat_and_vocal_sounds	Sustained sibilant sound of disapproval.
39	Low Mumble	throat_and_vocal_sounds	Indistinct vocalization of uncertainty or drowsiness.
40	Whispered Mumble	throat_and_vocal_sounds	Very quiet muddled speech sounds.
41	Humming	humming	Low continuous vocal tone often tuneful or absent-minded.
42	Purr	humming	Soft continuous vibrating hum showing contentment.
43	Resonant Hum	humming	Lower-pitched thoughtful hum.
44	Soft Hum	humming	Steady tone with closed lips in contentment.
45	Cough	coughing_and_sneezing	Sharp expulsion of air from lungs.
46	Coughing	coughing_and_sneezing	Series of forceful expulsions of air from lungs.
47	Hiccup	coughing_and_sneezing	Spasmodic contraction of diaphragm creating a 'hic' sound.
48	Hiccups	coughing_and_sneezing	Involuntary diaphragmatic spasm producing 'hic' sound.
49	Sniff	coughing_and_sneezing	Audible inhalation through the nose possibly sadness or restraint.
50	Snort	coughing_and_sneezing	Quick burst of air through the nose often disbelief.
51	Yawn	coughing_and_sneezing	Involuntary wide-mouthed breath showing tiredness.
52	Person Whistling Playfully	whistling	Lighthearted melodic whistle.
53	Person Whistling to Get Attention	whistling	Serious sharp whistle.
54	Sharp Whistle	whistling	Quick high-pitched attention grabber.
55	Soft Whistle	whistling	Gentle tuneful blowing of air.
56	Wolf Whistle	whistling	Two-note whistle expressing admiration.
57	Blowing a Kiss	mouth_and_lip_sounds	Brief airy sound of sending affection.
58	Kissing Noises	mouth_and_lip_sounds	Repetitive gentle lip compression for affection.
59	Kissing Sounds	mouth_and_lip_sounds	Soft repeated lip contact expressing affection.
60	Lip Smack	mouth_and_lip_sounds	Soft pop of parted lips indicating anticipation.
61	Smack One's Lips	mouth_and_lip_sounds	Light pop of lips often from dryness or taste anticipation.
62	Smacks Lips	mouth_and_lip_sounds	Repetitive lip popping after tasting or anticipating food.
63	Licking Sound	mouth_and_lip_sounds	Light quick contact of tongue on surface.
64	Spitting	mouth_and_lip_sounds	Forceful expulsion of saliva.
65	Sucking Noise	mouth_and_lip_sounds	Soft pulling sound created by suction.
66	Click One's Tongue	tongue_clicks	Short 'tch' sound by flicking tongue off palate.
67	Clicks Tongue	tongue_clicks	Distinct click by tapping tongue against the palate.
68	Tongue Click	tongue_clicks	Distinct clack made by tongue against palate.
69	Tsk	tongue_clicks	Clicking sound of reprimand or annoyance.
70	Chewing Noises	eating_and_drinking	Crunching or moist sounds from the mouth.
71	Drinking Noises	eating_and_drinking	Audible liquid intake with gulps or slurps.
72	Gulps	eating_and_drinking	Short audible swallow from nervousness or ingesting.
73	Nervous Gulp	eating_and_drinking	Audible swallow indicating anxiety.
74	Slurping Noises	eating_and_drinking	Wet sucking sound typically when consuming liquids.
75	Swallows	eating_and_drinking	Audible movement of liquid down the throat.
76	Scream	screams_and_shrieks	Loud high-pitched outburst of extreme emotion.
77	Shriek	screams_and_shrieks	Sharp piercing cry typically of fear.
78	Finger Snaps	hand_and_body_sounds	Brief sharp click made by friction of thumb and finger.
79	Hand Scratching Head	hand_and_body_sounds	Rubbing hair soft rustling of fingers on scalp.
80	Hand Slaps	hand_and_body_sounds	Solid impact sound of palm against a surface or another hand.
81	Slap Face	hand_and_body_sounds	Sharp impact sound of palm against face.
82	no_burst	—	No vocal burst present: ordinary speech, silence, background noise, or audio that does not contain a recognizable non-speech human vocalization. Also used for hand/body impact sounds that were removed from the taxonomy.

Recognized classes & folded (deprecated) classes

The model head is still 83-way (from the taxonomy), but 5 classes are folded to no_burst during training and masked at inference (never predicted), because they are hand gestures / a blown kiss — not audible vocal bursts, and they were confusing the classifier:

original index	class	mapped to
57	Blowing a Kiss	no_burst
78	Finger Snaps	no_burst
79	Hand Scratching Head	no_burst
80	Hand Slaps	no_burst
81	Slap Face	no_burst

So the recognized label space is 77 vocal-burst classes + no_burst. folded_classes.json gives the machine-readable mapping; class_descriptions.json marks the folded entries as DEPRECATED. (The four hand classes were already folded in V2 training; Blowing a Kiss is now folded too.)

Recognized burst classes: Breathy Giggle, Cackle, Childlike Giggle, Chuckle, Guffaw, Nervous Giggle, Snicker, Snorting Giggle, Convulsive Sob, Mournful Wail, Quiet Sob, Sobs, Trembling Whimper, Deep Breath, Deep Breathing, Fast Breathing, Heavy Breathing, Normal Breathing, Panting, Slow Breathing, Contented Sigh, Exasperated Sigh, Relief Sigh, Wistful Sigh, Fearful Gasp, Sharp Inhale, Surprised Gasp, Exhausted Groan, Frustrated Groan, Pain Moan, Pleasure Moan, Affirmative Grunt, Displeased Grunt, Effort Grunt, Ahem, Clears Throat, Growl, Gurgling, Hiss, Low Mumble, Whispered Mumble, Humming, Purr, Resonant Hum, Soft Hum, Cough, Coughing, Hiccup, Hiccups, Sniff, Snort, Yawn, Person Whistling Playfully, Person Whistling to Get Attention, Sharp Whistle, Soft Whistle, Wolf Whistle, Kissing Noises, Kissing Sounds, Lip Smack, Smack One's Lips, Smacks Lips, Licking Sound, Spitting, Sucking Noise, Click One's Tongue, Clicks Tongue, Tongue Click, Tsk, Chewing Noises, Drinking Noises, Gulps, Nervous Gulp, Slurping Noises, Swallows, Scream, Shriek.