SAGE: Semantic Audio Generative Encoder

SAGE is a compact variational autoencoder for stereo music at 44.1 kHz. A Swin Transformer V2 encoder and decoder operate on the STFT, and the latent is shaped by distilling the embeddings of LAION-CLAP. The model has 104.6M parameters and compresses a stereo waveform ×64 into a 16-channel latent at 86 frames per second (one latent frame per 512 samples).

• Paper: arXiv 2609.32755
• Code: github.com/francescobrigante/SAGE
• Project page: sage-music.pages.dev

Checkpoint

File	Size	SHA-256
SAGE_FTe992.ckpt (EMA weights, epoch 992)	402 MiB	dd87d01eaee88ca92f96c80ffe0e504a1d7cbc02e4271d8c48033df591d6ac99

The checkpoint holds the model configuration and the EMA weights of the decoder fine-tuning (phase 2, epoch 992), the model evaluated in the paper.

Usage

Install the code (Python 3.11, uv):

git clone https://github.com/francescobrigante/SAGE.git && cd SAGE
uv sync

import torch
from huggingface_hub import hf_hub_download
from sage import SAGE

ckpt = hf_hub_download("francescobrigante/SAGE", "SAGE_FTe992.ckpt")
codec = SAGE.from_checkpoint(ckpt)                               # CUDA if available
wav = torch.randn(1, 2, 10 * codec.sample_rate)                  # (B, 2, N) at 44.1 kHz

rec = codec.reconstruct(wav)                                     # encode + decode, same length as the input

padded, n = codec.pad(wav)                                       # right-pad to the model's frame grid
z = codec.encode(padded, deterministic=True)                     # posterior mean, (B, 16, frames)
y = codec.decode(z, target_length=padded.shape[-1])[..., :n]     # back to audio, input length

encode samples the latent unless deterministic=True. Training, evaluation and the commands that reproduce every result of the paper are in the repository.

Training data

About 10.5K hours of 44.1 kHz stereo music from three public corpora: the FMA-full training split, the MTG-Jamendo training partition (without the tracks of the Song Describer Dataset) and M4Singer. Two phases on 16 A100 GPUs: pretraining from scratch for 500 epochs, then decoder fine-tuning with the encoder frozen for 992 epochs (1,536 and 3,043 GPU-hours).

Intended use and limitations

SAGE is meant for research on music: a latent space for generative models (e.g. latent diffusion) and semantic analysis, and audio reconstruction. It is trained on music and solo singing; speech and general audio are out of domain. The weights are released under CC BY-NC-SA 4.0, following the non-commercial terms of part of the training data, and may not be used commercially.

Citation

@misc{brigante2026sage,
  title         = {{SAGE}: Semantic Audio Generative Encoder},
  author        = {Brigante, Francesco and Cerovaz, Luca and Marincione, Davide and Strano, Giorgio and
                   Zhou, Luca and Rodol{\`a}, Emanuele and Mancusi, Michele},
  year          = {2026},
  eprint        = {2609.32755},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2609.32755}
}