laion/audio-taxonomy-classifier

Audio taxonomy classifier

Tags an audio clip with a class from a hierarchical sound taxonomy of 4,416 nodes: 11 top-level groups, depth up to 5, 4,085 leaves. The model predicts 3,718 classes, the taxonomy nodes at depth 2 and 3 (deeper nodes are merged into their depth-3 ancestor), and separately one of the 11 top-level groups:

Speech · Music · Human non-speech · Animal / bioacoustics · Nature / environmental · Domestic / household · Urban / transport · Industrial / mechanical · Interface / electronic · Sound effects / designed sounds · Sports / recreation

The AudioSet ontology has 632 classes. This taxonomy is about seven times larger and much finer: for example, Speech/Noisy in-the-wild speech/Speech interrupted by electronic buzzer, or 100 classes under Music/Rock / metal-based music.

Intended use: cheap labelling of large audio collections for filtering, curation, balancing and search. Examples: measure how a collection splits over the taxonomy, cap over-represented groups, or find rare sound types. The top-level prediction is the reliable signal. The fine classes are best used as a ranked list (top-5).

Model

• Encoder: Dasheng-0.6B, frozen. The first 10 s of a clip are resampled to 16 kHz mono and zero-padded to 10 s. The frame features are mean-pooled to one 1280-d vector.
• Head (6.0M parameters): Linear 1280→1024, BatchNorm, GELU, dropout 0.2, then three linear outputs: 3,718 classes, 822 second-level nodes and 11 top-level groups.
• Readout: a class scores log P(top-level) + log P(second level) + log P(class), using its own ancestors, followed by a softmax over the 3,718 classes. The top-level group comes from the 11-way output.

Files

file	contents
head.safetensors	head weights; the metadata holds the config and the ordered class list
taxonomy.yaml	the full taxonomy (4,416 nodes, each with a description)
inference.py	loads Dasheng-0.6B and the head and classifies files

Usage

pip install torch dasheng soundfile scipy safetensors
python inference.py clip.flac

import torch
from inference import TaxonomyClassifier, load_audio

model = TaxonomyClassifier("head.safetensors")             # Dasheng weights download on first use
top, cls = model(torch.from_numpy(load_audio("clip.flac"))[None])
print(model.top_names[top.argmax()], model.classes[cls.argmax()])

Example output for a 2-s German sentence followed by an electronic buzzer:

  top-level: Speech (1.000)
  0.665  Speech/Noisy in-the-wild speech/Speech interrupted by electronic buzzer
  0.304  Speech/Instructional  -  tutorial speech
  0.012  Speech/Noisy in-the-wild speech/Speech preceding electronic test tone

Class names are taxonomy paths joined by /. A / inside a node name is written as -.

Training

The training clips come from public audio datasets and from web audio.

1. Caption-distilled labels. A large language model read the text caption of each of 232,728 clips and assigned the clip to a taxonomy node. The head was trained on the frozen Dasheng features with class-balanced cross-entropy (weights 1/√frequency) plus auxiliary top-level and second-level losses. The train, validation and test splits were made by source archive.
2. Audio-based relabelling (this release). Caption labels are noisy for non-speech sounds, so 80,159 more clips were sampled, weighted towards the eight groups other than Speech, Music and Animal, and towards clips where the first head was unsure. Three audio models labelled each clip independently: two contrastive audio-text models, each ranking the class names as text prompts, and Qwen2.5-Omni-7B, instructed to pick a top-level group. For those eight groups a clip was kept only if all three models agreed on its top-level group (14,981 clips). Its class label is then the top class of the more confident contrastive model. In Speech, Music and Animal, the first head's own label was kept when its top-level probability was ≥ 0.6 (17,639 clips). The head was then fine-tuned from stage 1 on these 32,620 clips for 60 epochs, using the same loss, AdamW at lr 1e-3 and a cosine schedule.

The benchmark datasets used for evaluation below were excluded from the relabelling sample, by source dataset. They were also excluded from the stage-1 data, except VocalSound and TAU Urban Acoustic Scenes 2022. Those two were not filtered there, but any overlap is small: stage 1 sampled 232,728 clips from a much larger pool. Copies of benchmark recordings that reached the pool through other sources (re-uploads) were not screened out.

Evaluation

Top-level accuracy on public benchmark test sets. Each benchmark class has a fixed set of acceptable top-level groups, chosen before scoring (1–3 groups; ESC-50 "crackling fire", for example, accepts Nature, Domestic or Sound effects). A prediction counts as correct if its group is in that set. For FSD50K (multi-label), the set is the union over the clip's labels. "Leaf-derived" reads the top-level group off the best class instead of the 11-way output.

benchmark	clips	top-level acc. (95 % CI)	leaf-derived
CommonVoice (test)	2,500	99.9 (99.6–100.0)	99.9
CREMA-D	1,500	100.0 (99.7–100.0)	100.0
GTZAN	999	99.9 (99.4–100.0)	99.8
BirdCLEF 2025	1,500	96.0 (94.9–96.9)	96.2
NSynth (test)	2,000	76.3 (74.4–78.2)	78.4
ESC-50	2,000	76.2 (74.3–78.0)	78.7
FSD50K (eval)	2,000	75.0 (73.1–76.8)	76.5
VocalSound	2,500	73.6 (71.8–75.3)	75.3
UrbanSound8K	835	73.2 (70.1–76.1)	72.7
macro, 9 benchmarks	15,834	85.6	86.4
TAU Urban Acoustic Scenes 2022 (scenes, not events)	1,006	63.8	56.4

• Calibration: the expected calibration error of the top-level probability is 0.066 over the 9 benchmarks.
• For comparison, Qwen2.5-Omni-7B, instructed to pick one of the 11 groups, reaches 87.6 % macro on the same clips. It costs far more per clip.
• Class level: the predicted class matches the benchmark label (by a name pattern per benchmark class) for ESC-50 47.0 % (top-5: 66.8 %), GTZAN 69.4 % (86.1 %), VocalSound 52.0 % (75.7 %), UrbanSound8K 29.5 % (41.9 %) and NSynth 24.9 % (53.2 %).
• On a random 3 % of its own relabelled training set, held out, the head agrees with the consensus labels on 96.0 % of top-level groups and 48.6 % of classes.

Limitations

• Fine classes are noisy. No large ground-truth set exists at that level, so use them as a ranked list.
• Only the first 10 s of a clip are used, and a clip gets a single path. Mixed content (speech over music, film audio) gets one label.
• Top-level accuracy is lowest on everyday sound events, non-speech vocal sounds and isolated instrument notes (ESC-50, FSD50K, UrbanSound8K, VocalSound, NSynth: 73–76 %). On acoustic scenes it is lower still (TAU 63.8 %).
• The taxonomy was grown from captions of web audio, so the depth of its branches follows what is common on the web, not a designed balance.

Licence and attribution

Apache-2.0. The encoder is Dasheng-0.6B (Apache-2.0). The Human non-speech vocal-burst classes in taxonomy.yaml were merged from LAION-AI/voice-taxonomies (CC-BY-4.0).