swillsoftware/describesong-tagger

describesong sound tagger — ONNX export

Replaces the AudioSet tagger (Xenova/ast-finetuned-audioset-10-10-0.4593) for the two questions the site actually asks. Measured on Led Zeppelin's 30-second iTunes preview, which William confirmed by ear contains guitar and singing:

	AST	CED-base	this
acoustic guitar	0.032	0.052	0.571
electric guitar	0.003	0.000	0.602
guitar	0.053	0.075	0.514
drums	0.066	0.074	0.635
bass	0.038	—	0.611
voice	0.113	0.139	0.907

AST and CED are AudioSet models — 527 general classes (speech, dogs, sirens) with instruments as a handful of labels. These are music taggers.

Three heads on one embedding — AST is gone

Measured on all 1,408 LRCLIB-labelled tracks, the voice head at 0.50 (MEAN over patches) scores 0.868 balanced against AST's vocal classes' 0.824. So the AudioSet tagger is removed from both the browser and the indexer, and this embedding carries all three questions:

head-instrument.onnx   1280 -> 40 sigmoid   which instruments (MAX over patches, threshold 0.4)
head-voice.onnx        1280 -> 2 softmax    is anyone singing (MEAN over patches; 0.5 = midpoint)
head-gender.onnx       1280 -> 2 softmax    male / female singer (MEAN; only meaningful when voice >= 0.5)

All verified against Essentia's own pipeline: instrument 4.2e-07, voice 9.7e-08, gender 1.2e-07. The site uses voice >= 0.50 for "instrumental" search and voice < 0.30 to skip transcription (a skipped song is unrecoverable, a transcribed instrumental costs ten seconds).

Files (19.7 MB total — upload as-is)

effnet.onnx              16.0 MB   Discogs-EffNet, mel -> 1280-d embedding
head-instrument.onnx      2.7 MB   MTG-Jamendo instrument, 1280 -> 40 (sigmoid)
head-voice.onnx           0.5 MB   voice/instrumental, 1280 -> 2 (softmax)
head-gender.onnx          0.5 MB   male/female singer, 1280 -> 2 (softmax)
melbank_96x257.npy       96.4 KB   the filterbank, extracted from Essentia
instrument-classes.json            the 40 class names, in order
voice-classes.json                 ['instrumental', 'voice']

Converted with tf2onnx opset 17 from Essentia's published graphs. Both heads read the SAME embedding, so it is one forward pass through effnet plus two trivial matmuls — not three models.

THE INPUT IS A MEL SPECTROGRAM, NOT AUDIO

effnet.onnx input is serving_default_melspectrogram, shape (64, 128, 96).

The batch is PINNED at 64. Same class of trap as the MuLan export, where a baked-in shape constant pinned the batch to 1 and silently returned wrong vectors for windows 2 and 3. Feed exactly 64 patches, zero-padded, and read back only the rows you filled.

Every one of these was established by measurement against Essentia's own pipeline, not from documentation:

16 kHz mono
pad 256 zeros at the FRONT          (Essentia's startFromZero=False:
                                     the first frame is centred on sample 0)
frames of 512, hop 256
window: SYMMETRIC Hann              (np.hanning(512); periodic is wrong,
                                     maxdiff 2.7e-2 instead of 2.5e-6)
power spectrum: |rfft|**2
mel: spec @ melbank.T               (96 bands, 0-8000 Hz, slaneyMel warping,
                                     unit_tri normalisation, power, linear
                                     weighting)
log10(1 + 10000 * mel)
patches of 128 frames, hop 62       (62, not 64 or 128 — it is what Essentia
                                     uses and it decides patch alignment)
batch 64, zero-padded

The filterbank is shipped rather than recomputed because the slaneyMel + unit_tri combination is fiddly to reproduce and getting it wrong is silent. It was extracted by probing Essentia's MelBands with a unit impulse per FFT bin, which is exact.

Read the heads as: instrument = max over patches per class (the rule tagger.tag() already uses); voice = MEAN over patches (it is a softmax over a 2-class decision about the whole track, not an event to be spotted once).

Verified

From raw audio with Essentia nowhere in the path, against Essentia's own end-to-end pipeline on the same file:

mel frames      maxdiff 2.9e-06
embedding       maxdiff 1.5e-06
instrument head maxdiff 4.2e-07
voice head      maxdiff 9.7e-08

Still to do before this can ship

• Browser parity: run it under ORT-web and check against these numbers. The MuLan export's two traps both apply — fp16 returning NaN on some providers, and batching silently producing wrong rows.
• A 512-point FFT in web/stft.js (it currently only does 2048).
• Re-tagging: ~700 preview rows the indexer can redo itself; 2,280 scanned rows need their owners to re-scan.
• EVENT_MIN was calibrated against AST's numbers and will need recalibrating.

Licence

Converted from Essentia's published models (https://essentia.upf.edu/models/), which are CC BY-NC-SA 4.0 — non-commercial. The export inherits that. Same question as MuQ-MuLan's CC-BY-NC-4.0 weights, which describesong.com's licence page already sets out: whether a non-commercial weights licence reaches the numbers a model produces is unsettled, and this repo is not where it gets decided.

Underlying work: MTG-Jamendo instrument model and voice/instrumental classifier on Discogs-EffNet embeddings, by the Music Technology Group, UPF.

Browser parity — RUN AND MEASURED, in Chrome, against these numbers

mel512.js in this folder is the browser-side mel (512-point FFT, the constants above). Verified end to end from a wav through ORT-web:

ORT	provider	worst diff	result
1.20.1	webgpu	2.93e-1	WRONG — acousticguitar 0.4683, voice 0.6262
1.20.1	wasm	4.91e-5	ok, 36 ms/patch
1.23.0	webgpu	4.96e-5	ok — acousticguitar 0.5713, voice 0.9066

web/worker.js pins onnxruntime-web@1.20.1, which is the broken one. It is the EMBEDDING model (effnet) that is wrong on 1.20.1's WebGPU, not the heads — isolated by running each on a different provider. No error, no warning, just different numbers: the same failure mode as the MuLan batch-shape trap.

Safest wiring: import 1.23 for the tagger and leave MuLan on 1.20.1, rather than moving MuLan to a version its export has never been checked against. Two ORT copies cost one extra wasm download, which is cheap next to re-verifying 9,000 vectors.

Cost is per BATCH, not per patch

The batch is pinned at 64, so 4 patches cost the same as 64. Measured:

29 patches (one batch)   webgpu 1.23    219 ms
29 patches (one batch)   wasm   1.20   1061 ms

A 4-minute track is ~232 patches = 4 batches ~= 0.9 s on WebGPU, against the ~1.6 s/track the scan already costs. Do NOT subsample patches to go faster: it is free on the batch and it costs signal — 16 of the 29 patches drops acousticguitar from 0.5713 to 0.4683.