georgeven/songmae-large-32x4

SongMAE-Large 32x4

SongMAE is a masked autoencoder pretrained for high-resolution birdsong representation learning. This Large checkpoint uses patches spanning 32 mel bins and 4 spectrogram time bins, producing one embedding every 20 ms.

• SongMAE model collection
• SongMAE: A bioacoustic encoder for birdsong

The model was pretrained for 500,000 steps on the XCL subset of BirdSet, derived from 528,434 Xeno-Canto recordings totaling 7,562 hours. Downstream evaluation taxa were removed before the final 95/5 train/validation split.

Usage

pip install "transformers>=4.36" "librosa>=0.10" safetensors

import numpy as np
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "georgeven/songmae-large-32x4",
    trust_remote_code=True,
).to("cuda")

embeddings = model.embed_file("bird.wav")
print(embeddings["clip_embedding"].shape)    # (3072,)
print(embeddings["token_embeddings"].shape) # (time, 3072)
print(embeddings["token_grid"].shape)       # (time, 4, 768)
print(embeddings["timestamps_ms"].shape)    # (time, 2)

np.savez("bird_songmae_32x4.npz", **embeddings)

Remove .to("cuda") to run on CPU. embed_audio(waveform, sample_rate) accepts a mono NumPy waveform directly.

Raw forward() accepts normalized log-mel tensors shaped (batch, 1, 128, 1000) and returns the encoder sequence as last_hidden_state.

Input and output contract

• Input audio is converted to mono 32 kHz.
• Spectrograms use 128 mel bins, a 1024-sample FFT, a 160-sample hop, 20 Hz minimum frequency, and decibels relative to each recording's maximum power.
• Spectrogram values use the fixed pretraining mean -58.69395667478683 and standard deviation 20.184339052439707.
• Audio is encoded in bounded 5-second contexts. The final context is padded after normalization and excluded from returned embeddings.
• Each 20 ms output contains four 768-dimensional frequency-patch vectors. token_embeddings concatenates these into 3072 dimensions; clip_embedding is their mean across time.

Intended use and limitations

The checkpoint is intended as a frozen encoder for birdsong retrieval, visualization, clustering, and downstream probes. It is not a species classifier and does not generate audio. Field conditions and taxa unlike the pretraining distribution may require validation or adaptation. Long recordings are processed as independent 5-second contexts, so representations do not attend across context boundaries.

Provenance

• Parameters: 98,167,553
• Source run: xcl_large_500k_p32x4_c010
• Source checkpoint: model_step_499999.pth
• Source SHA-256: bcad9ac2544f281a83524da88b037207afb52ef2c6dafa924193ba6b83d3f964
• Source code: https://github.com/georgevenven/SongMAE