mlboydaisuke/AST-AudioSet-ExecuTorch

AST audio event classification — ExecuTorch

Ten seconds of sound in, 527 AudioSet labels out: speech, music, a dog, a door, an engine. The shelf had speech recognition and speaker identity; this is the one that says what a sound is.

• Source: MIT/ast-finetuned-audioset-10-10-0.4593 — 86M parameters, Audio Spectrogram Transformer
• License: bsd-3-clause
• Input: input_values [1, 1024, 128] fp32 — log-mel filterbank [1, 1024, 128] fp32 — torchaudio.compliance.kaldi.fbank at 16000 Hz with 128 mel bins, padded or trimmed to 1024 frames (10.24 s), then normalised with mean=-4.2677393 std=4.5689974. ASTFeatureExtractor does exactly this
• Output: logits [1, 527] — AudioSet labels, multi-label: apply sigmoid, not softmax

Variants

build	file	size (MB)	Mac median (ms)*	top-1 vs eager	worst probability shift
fp32	audiocls_audioset_xnnpack_fp32.pte	346.6	285.0	8 of 8	0.0000
fp16	audiocls_audioset_xnnpack_fp16.pte	173.9	545.1	8 of 8	0.0014
int8 (dynamic)	audiocls_audioset_xnnpack_int8.pte	90.9	267.3	8 of 8	0.0160
Core ML (fp16, iOS)	audiocls_audioset_coreml_all.pte	173.7	74.6	8 of 8	0.0033

*Mac arm64, single process, median of 10, one 10.24 s clip. PyTorch eager fp32 on the same machine is 122.7 ms. Core ML at 74.6 ms is 1.6x that; int8 at 267.3 ms is the fastest portable build and a quarter of the fp32 file. fp16 is slower than fp32 here (545.1 ms) — XNNPACK emulates it — and only earns its place by halving the file.

What the classifier actually says

The test clips are speech, and every build puts Speech in the top five on all 8 of them. The top-1 label matches eager on 8 of 8, the top-five sets overlap 40 of 40, and no sigmoid probability moves by more than the figure in the table.

The distance the error has to cover is printed too: the gap between the winning logit and the runner-up is at least 3.09 on these clips, which every build's shift is far inside. Agreement alone would not show this — a build returning a constant vector would agree with a broken reference on every clip — so the label check is there as well.

The features are the caller's job, and the recipe is exact

AST's front end is torchaudio.compliance.kaldi.fbank: a Kaldi-compatible filterbank with its own windowing and edge handling. Reimplementing it inside the graph would be a second model's worth of work for a transform transformers runs in two lines, so the graph starts at the spectrogram. The recipe is read off the model's own preprocessor rather than written from memory:

from transformers import AutoFeatureExtractor
extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
inputs = extractor(waveform, sampling_rate=16000, return_tensors="pt")["input_values"]

Getting it wrong does not throw. It shifts every probability.

The output is multi-label: apply sigmoid, not softmax. A ten-second clip can be speech and music and a car at once, which is the point of AudioSet.

Conversion

python convert/export_audiocls.py audioset
python convert/check_audiocls.py audioset int8

(conversion scripts: executorch-models)