dev-e/panns-cnn14-16k-coreml

PANNs CNN14 (16 kHz), Core ML conversion

A Core ML conversion of the PANNs CNN14 16 kHz AudioSet tagging model, used by VERITA as an optional second sound-event detector next to Apple's built-in classifier. The app downloads this folder file by file and checks every file's SHA-256 before using it.

Source and attribution

• Model and weights: Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, Mark D. Plumbley, "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", IEEE/ACM TASLP 2020 (arXiv:1912.10211).
• Weights file used: Cnn14_16k_mAP=0.438.pth from Zenodo record 3987831, licensed CC BY 4.0; SHA-256 e2ee543a27919542c2ea03eabaa70b24dcd4e6c8e05621de6b67a94e4c5058e6 (358,668,570 bytes).
• Model code: qiuqiangkong/audioset_tagging_cnn (MIT).
• Class names (labels.txt): the 527 AudioSet classes in the model's output order, from class_labels_indices.csv of that repository (AudioSet, Google, CC BY 4.0).

This conversion is licensed CC BY 4.0, like the original weights. If you use it, credit the PANNs authors as above.

Changes made to the original

The weights are the original ones; only their format and precision changed:

1. The PyTorch model (Cnn14, sample rate 16000, window 512, hop 160, 64 mel bins, 50–8000 Hz, 527 classes) was put in inference mode and wrapped to return only clipwise_output.
2. It was traced and converted with coremltools to an ML Program (Cnn14_16k.mlpackage, minimum macOS 13), with float16 weights and compute precision. Batch normalisation was folded into the convolutions by the converter.
3. Check after conversion: on the same input windows, Core ML scores differ from PyTorch scores by at most 0.0006.

Nothing was retrained or fine-tuned.

Interface

• Input waveform: float32 [batch, 48000] (batch 1 to 64), mono audio at 16 kHz, 3 s per row, in [-1, 1].
• Output scores: [batch, 527], one sigmoid score per AudioSet class (order of labels.txt).

VERITA runs it on 3 s windows every 1.5 s and maps a reviewed subset of AudioSet classes to its own sound classes; a score of 0.5 or more makes a detection, which a person then decides on.

Files

File	Bytes	SHA-256
Cnn14_16k.mlpackage/Manifest.json	617	f1b114c52d5ed2c12cbf9b2bc554dd04dcf14c54b431bb728d6613e6eeb8bb6e
Cnn14_16k.mlpackage/Data/com.apple.CoreML/model.mlmodel	33,882	8bd1a28ec9c0a12b4da928b99bacd83699e1e3cc0951e18258bfcd3d2e7007a1
Cnn14_16k.mlpackage/Data/com.apple.CoreML/weights/weight.bin	162,052,894	38cdbed0c4696e576eb01aeae276f7918ccdbb6ab9755703867fe46fb43a931d
labels.txt	6,337	83e2e4fd16cbe86f94f3fe23954a4d302c013e20e7981b5ce7466e50126a1e3d

Folder hash used by VERITA's model registry (SHA-256 of the sorted lines <path>\t<file SHA-256>\n over the four files above): 1243a933f9dac6931e24256daa474d444d26a1f319f4385bbc29c46a8b945bc0.

Limitations

Measured in VERITA's own benchmark (44 CC0 sound effects mixed into generated speech), at a 0.5 score: 23 % of the effects caught on its own, and no effect caught that Apple's classifier missed; shouting was not recognised. AudioSet tags describe sounds, not facts: every detection is a suggestion for a person to check.