stratus-labs/nocturne-v1.2-teacher

Nocturne v1.2 (Teacher) — non-bird bioacoustic species recognition

Nocturne identifies the taxa bird-focused models leave out: insects, amphibians, non-bird mammals and reptiles. v1.2 adds Neotropical anurans and, more importantly, fixes two defects in how v1 was supervised and selected.

v1.2 is the first model in this family to beat the v1 baseline on the frozen benchmark, on both metrics, measured on the identical 13,710-clip held-out test set with identical name-aligned class columns.

held-out test (calibrated per-class thresholds)	v1 teacher	v1.2	delta
macro-F1 over all 2,182 core classes	0.148766	0.151051	+0.002286
mAP (macro)	0.149849	0.154110	+0.004262
macro-F1 over the 732 classes with test positives	0.4435	0.4503	+0.0068
top-1 accuracy	0.4411	0.4384	−0.0027

Both improvements exceed the noninferiority margin we pre-committed before scoring (0.00169 macro-F1, 0.00152 mAP), derived from a paired bootstrap over the test rows. They are improvements, not noise.

What actually produced the gain — and what did not

We ran four attempts at v1.2. Three failed to beat v1, and saying which is more useful than the headline:

attempt	change	core mAP	verdict
v1.2	+ AnuraSet weak labels (recording-level)	0.1276	worse — the model learned site signatures, not species
v1.2b	+ AnuraSet strong labels, medium-quality subset	0.1486	no better than v1
v1.2c	+ fixed label parser (all call qualities)	0.1474	the label fix alone bought nothing
v1.2 (this model, internally v1.2d)	fixed parser + multi-hot targets + mAP selection	0.1541	passes

The dataset was never the bottleneck. Two supervision bugs were:

1. One positive per training row. The loader emitted one row per (clip, species) pair with a single positive, so a clip containing two species was trained twice, each time teaching the model the other species was absent. Training now uses one multi-hot row per clip. This is the change that produced the gain.
2. Checkpoint selection by F1 at a fixed 0.5 threshold. The shipped metric is threshold-calibrated, so selection was optimising the wrong quantity. Selection is now by validation mAP. This model's best epoch was 18 of 40 — the old rule would have shipped a much later, worse checkpoint.

Where it does not improve

On a held-out monitoring site never seen in training (AnuraSet INCT41), v1.2 reaches top-1 0.138. That is roughly ten times better than the weak-label attempt (0.005), but still well below v1's 0.544. Adding AnuraSet has not bought generalisation to a new recording site. If your use case is deploying to an unseen site for anuran monitoring specifically, v1 may still serve you better, and we would rather say so than bury it. Only 3–5 species are scorable at that site, so treat that number as a signal, not a ranking.

Model

Architecture	Audio Spectrogram Transformer, from MIT/ast-finetuned-audioset-10-10-0.4593
Head	Linear over 2,196 species (multi-label BCE)
Input	10-second mono waveform @ 16 kHz → 128-band log-mel
Params	~86 M
Selected	epoch 18 of 40, by validation mAP
Trained on	1× NVIDIA GB10 (DGX Spark), torch 2.13.0, transformers 5.16.1
License	weights CC-BY-4.0, code Apache-2.0

Training data

86,703 multi-hot training rows (one per audio file) from InsectSet459, the non-bird portion of iNat Sounds 2024, and AnuraSet strong labels (per-vocalisation time bounds, all call qualities, ≥50 % window overlap). Note the row count is files, not the source datasets' clip totals — an earlier version of this card conflated the two and overstated it.

Evaluation protocol

One frozen harness, soundscape/benchmark.py, scores every checkpoint identically: InsectSet459 + iNat held-out test, one multi-hot row per audio file, per-class thresholds calibrated on validation (grid 0.02→0.94, default 0.3 where a class has no validation positives), macro-averaged over the 2,182 v1 core classes matched by name so models with different vocabularies remain comparable. The harness is validated by reproducing v1's published numbers to four decimals before any comparison is trusted.

macro_f1 averages over all 2,182 columns, but only 732 have positives in the test split, so ~1,450 columns contribute exactly zero regardless of model quality. The companion figure over scorable classes only (0.4503) is the more interpretable one. Both are reported here; neither is cherry-picked.

Files

file	purpose
best.pt	selected weights (epoch 18), torch.save state dict
model.py	standalone strict loader (handles the transformers 5.8→5.16 parameter rename)
vocab.json	species → class index (2,196)
thresholds.json	per-class calibrated thresholds
eval_report.json	full benchmark output, core and anuran slices
config.yaml, run_manifest.json	exact recipe, seed, code hash, library versions

Usage

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("stratus-labs/nocturne-v1.2-teacher")
sys.path.insert(0, path)
from model import load_nocturne, predict_file

model, vocab, thresholds = load_nocturne(path)      # strict — raises on any key mismatch
for species, score in predict_file(model, "clip.wav", vocab, top_k=5):
    print(f"{score:.3f}  {species}")

Never load these weights with strict=False. A silent partial load leaves the backbone at AudioSet initialisation and produces plausible-looking nonsense; that bug cost us a full round of invalid benchmarks.

Limitations

• Long-tail. 2,196 classes, 732 with test positives. Rare species are effectively unvalidated.
• Not for birds. Use BirdNET or Perch.
• New-site generalisation for anurans is worse than v1 (see above).
• Not for legal or conservation decisions without field verification.
• Dual-use. Acoustic localisation of rare taxa can aid illegal collection; consider redacting precise GPS in public datasets built with this model.

Related

• Baseline: Nocturne v1 teacher · distilled v1 mini
• Sibling family: Radiance (Sentinel-1 SAR)
• All releases: huggingface.co/stratus-labs