georgeven/songmae-micro-32x4

SongMAE-Micro 32x4

SongMAE is a masked autoencoder pretrained for high-resolution birdsong representation learning. This Micro checkpoint uses patches spanning 32 mel bins and 4 spectrogram time bins, producing one embedding every 20 ms.

• SongMAE model collection
• SongMAE: A bioacoustic encoder for birdsong

The model was pretrained for 500,000 steps on the XCL subset of BirdSet, derived from 528,422 Xeno-Canto recordings totaling 7,562 hours. We first made the 95/5 training/validation split, then removed recordings from downstream evaluation birds from the training set.

Usage

pip install "transformers>=4.36" "librosa>=0.10" safetensors

import numpy as np
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "georgeven/songmae-micro-32x4",
    trust_remote_code=True,
).to("cuda")

embeddings = model.embed_file("bird.wav")
print(embeddings["clip_embedding"].shape)    # (512,)
print(embeddings["token_embeddings"].shape) # (time, 512)
print(embeddings["token_grid"].shape)       # (time, 4, 128)
print(embeddings["timestamps_ms"].shape)    # (time, 2)

np.savez("bird_songmae_micro_32x4.npz", **embeddings)

Remove .to("cuda") to run on CPU. embed_audio(waveform, sample_rate) accepts a mono NumPy waveform directly.

Raw forward() accepts normalized log-mel tensors shaped (batch, 1, 128, 1000) and returns the encoder sequence as last_hidden_state.

Input and output contract

• Input audio is converted to mono 32 kHz.
• Spectrograms use 128 mel bins, a 1024-sample FFT, a 160-sample hop, 20 Hz minimum frequency, and decibels relative to each recording's maximum power.
• Spectrogram values use the fixed pretraining mean -58.69395667478683 and standard deviation 20.184339052439707.
• Audio is encoded in bounded 5-second contexts. The final context is padded after normalization and excluded from returned embeddings.
• Each 20 ms output contains four 128-dimensional frequency-patch vectors. token_embeddings concatenates these into 512 dimensions; clip_embedding is their mean across time.

Intended use and limitations

The checkpoint is intended as a compact frozen encoder for birdsong retrieval, visualization, clustering, and downstream probes. It is not a species classifier and does not generate audio. Field conditions and taxa unlike the pretraining distribution may require validation or adaptation. Long recordings are processed as independent 5-second contexts, so representations do not attend across context boundaries.

Provenance

• Parameters: 1,674,305
• Source run: xcl_micro_500k_p32x4_c010
• Source checkpoint: model_step_499999.pth
• Source SHA-256: 11e5a4969f9218b2c76647d585c2ac82e1196172b91514d4800f5d08ecbd0444
• Source code: https://github.com/georgevenven/SongMAE