SonicEmbed

SonicEmbed is an EmbeddingGemma 2 fine-tune that puts text and music in one 768-dimensional space from a single model that still works for general text. On Song Describer, it beats LAION-CLAP's music checkpoint. The vision tower of the base model isn't included in this checkpoint.

Text-to-music search, Song Describer	EmbeddingGemma 2	CLAP music	SonicEmbed
R@10, three 10 s clips per track	14.3	33.5	40.4
R@10, one 10 s clip	11.4	27.5	36.7

Usage

pip install sentence-transformers librosa pillow torchvision

Pillow and torchvision are required even though images aren't used: the base EmbeddingGemma 2 processor builds its image and video parts when it loads.

import librosa
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jaisawhney/SonicEmbed")


def embed_track(path):
    """The mean of three 10 s clips, at 25 %, 50 % and 75 % of the track."""
    wave, sr = librosa.load(path, sr=16000, mono=True)
    size = 10 * sr
    starts = [min(max(int(p * len(wave)) - size // 2, 0), max(len(wave) - size, 0)) for p in (0.25, 0.5, 0.75)]
    clips = [{"array": wave[s:s + size], "sampling_rate": sr} for s in starts]
    mean = model.encode(clips, normalize_embeddings=True).mean(0)
    return mean / np.linalg.norm(mean)


track = embed_track("song.flac")
query = model.encode("calm piano", prompt_name="query", normalize_embeddings=True)
print(query @ track)

• Prompts: use prompt_name="query" for search text, and prompt_name="document" for captions you search from a clip. Audio takes no prompt.
• Links count as audio: a text input that is only a link to an .mp3, .wav, .ogg, .flac or .aac file is fetched as audio.
• In-memory audio must already be 16 kHz: sampling_rate is ignored, so resample first.
• Only the first 30 s of audio is used: anything longer is truncated. To cover a whole track, average clips as above.

Results

Training data was checked and filtered for Song Describer tracks. SonicEmbed, EmbeddingGemma 2, and CLAP were scored with the same code in the same run. CLAP is LAION-CLAP's music_audioset_epoch_15_esc_90.14.pt.

Song Describer

Clips per track	Model	R@1	R@5	R@10	Median rank
Three 10 s	SonicEmbed	9.5	29.4	40.4	16
	CLAP music	5.0	21.8	33.5	24
	EmbeddingGemma 2	2.7	8.9	14.3	81
One 10 s	SonicEmbed	8.4	23.8	36.7	20.5
	CLAP music	4.9	16.8	27.5	29
	EmbeddingGemma 2	1.4	6.1	11.4	89

Song Describer has 1,106 human-written captions for 706 tracks. Each caption is used as a query to retrieve its track. With 1,106 captions, R@10 has a margin of error of about ±3 points. Scores can differ by a few tenths to about 1/2 a point between machines.

General text

MTEB(eng, v2), Mean(Task)
EmbeddingGemma 2, as reported on its model card	68.46
SonicEmbed, measured with eval/mteb_tasks.py	64.66

The two figures come from different runs, whose setups may differ. Expect some loss on general text.

Limitations

SonicEmbed was trained on English-only captions, and other languages are untested.

Training

Fine-tuned with Sentence Transformers on 1,234,845 caption and clip pairs from IAMD, minus its non-commercial recordings, with MusicCaps making up 10 % of the pairs. The result was blended with the original weights to recover some general text quality, with the ratio chosen on a held-out split of the training data.

IAMD and MusicCaps were checked for Song Describer tracks, and any items found were excluded from training (list).

Reproducing the numbers

eval/ has the Song Describer, CLAP, and MTEB scripts, with instructions.

Licence and credits

The weights are released under Apache-2.0. Recordings used in training that require attribution are credited in CREDITS.csv, where identifier is the Internet Archive item ID.