georgeven/songmae-base-32x1

SongMAE-Base 32x1

SongMAE is a masked autoencoder pretrained for high-resolution birdsong representation learning. This canonical Base checkpoint uses patches spanning 32 mel bins and 1 spectrogram time bin, producing one embedding every 5 ms.

• SongMAE model collection
• SongMAE: A bioacoustic encoder for birdsong

The model was pretrained for 500,000 steps on the XCL subset of BirdSet, derived from 528,422 Xeno-Canto recordings totaling 7,562 hours. We first made the 95/5 training/validation split, then removed recordings from downstream evaluation birds from the training set.

Usage

pip install "transformers>=4.36" "librosa>=0.10" safetensors

import numpy as np
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "georgeven/songmae-base-32x1",
    trust_remote_code=True,
).to("cuda")

embeddings = model.embed_file("bird.wav")
print(embeddings["clip_embedding"].shape)    # (1536,)
print(embeddings["token_embeddings"].shape) # (time, 1536)
print(embeddings["token_grid"].shape)       # (time, 4, 384)
print(embeddings["timestamps_ms"].shape)    # (time, 2)

np.savez("bird_songmae_base_32x1.npz", **embeddings)

Remove .to("cuda") to run on CPU. embed_audio(waveform, sample_rate) accepts a mono NumPy waveform directly.

Raw forward() accepts normalized log-mel tensors shaped (batch, 1, 128, 1000) and returns the encoder sequence as last_hidden_state.

Input and output contract

• Input audio is converted to mono 32 kHz.
• Spectrograms use 128 mel bins, a 1024-sample FFT, a 160-sample hop, 20 Hz minimum frequency, and decibels relative to each recording's maximum power.
• Spectrogram values use the fixed pretraining mean -58.69395667478683 and standard deviation 20.184339052439707.
• Audio is encoded in bounded 5-second contexts. The final context is padded after normalization and excluded from returned embeddings.
• Each 5 ms output contains four 384-dimensional frequency-patch vectors. token_embeddings concatenates these into 1536 dimensions; clip_embedding is their mean across time.

Intended use and limitations

The checkpoint is intended as a frozen encoder for birdsong retrieval, visualization, clustering, and downstream probes. It is not a species classifier and does not generate audio. Field conditions and taxa unlike the pretraining distribution may require validation or adaptation. Long recordings are processed as independent 5-second contexts, so representations do not attend across context boundaries.

Provenance

• Parameters: 14,889,409
• Source run: xcl_base_500k_p32x1_c005
• Source checkpoint: model_step_499999.pth
• Source SHA-256: 21550f6b22d5d062be686ceceebff559b8405d816628151c35e683221e465074
• Source code: https://github.com/georgevenven/SongMAE