georgeven/songmae-large-32x1

SongMAE-Large 32x1

SongMAE is a masked autoencoder pretrained for high-resolution birdsong representation learning. This canonical Large checkpoint uses patches spanning 32 mel bins and 1 spectrogram time bin, producing one embedding every 5 ms.

• SongMAE model collection
• SongMAE: A bioacoustic encoder for birdsong

The model was pretrained for 500,000 steps on the XCL subset of BirdSet, derived from 528,434 Xeno-Canto recordings totaling 7,562 hours. Downstream evaluation taxa were removed before the final 95/5 train/validation split.

Usage

pip install "transformers>=4.36" "librosa>=0.10" safetensors

import numpy as np
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "georgeven/songmae-large-32x1",
    trust_remote_code=True,
).to("cuda")

embeddings = model.embed_file("bird.wav")
print(embeddings["clip_embedding"].shape)    # (3072,)
print(embeddings["token_embeddings"].shape) # (time, 3072)
print(embeddings["token_grid"].shape)       # (time, 4, 768)
print(embeddings["timestamps_ms"].shape)    # (time, 2)

np.savez("bird_songmae_32x1.npz", **embeddings)

Remove .to("cuda") to run on CPU. embed_audio(waveform, sample_rate) accepts a mono NumPy waveform directly.

Raw forward() accepts normalized log-mel tensors shaped (batch, 1, 128, 1000) and returns the encoder sequence as last_hidden_state.

Input and output contract

• Input audio is converted to mono 32 kHz.
• Spectrograms use 128 mel bins, a 1024-sample FFT, a 160-sample hop, 20 Hz minimum frequency, and decibels relative to each recording's maximum power.
• Spectrogram values use the fixed pretraining mean -58.69395667478683 and standard deviation 20.184339052439707.
• Audio is encoded in bounded 5-second contexts. The final context is padded after normalization and excluded from returned embeddings.
• Each 5 ms output contains four 768-dimensional frequency-patch vectors. token_embeddings concatenates these into 3072 dimensions; clip_embedding is their mean across time.

Intended use and limitations

The checkpoint is intended as a frozen encoder for birdsong retrieval, visualization, clustering, and downstream probes. It is not a species classifier and does not generate audio. Field conditions and taxa unlike the pretraining distribution may require validation or adaptation. Long recordings are processed as independent 5-second contexts, so representations do not attend across context boundaries.

Provenance

• Parameters: 98,645,249
• Source run: xcl_large_500k_p32x1_c005
• Source checkpoint: model_step_499999.pth
• Source SHA-256: f12a1c0d7848a5875fd1c6c72492e6362a4c9ef6aa7036b3268f71d565fe3865
• Source code: https://github.com/georgevenven/SongMAE