omura-embed-audio

A small linear adapter head trained on top of frozen laion/larger_clap_general (CLAP) audio embeddings, used by Omura for natural-language audio search over the Walrus protocol's audio corpus.

Important — what this repo is and isn't:

• This repo contains only the trained adapter head (omura_clap_head.pt, ~1 MB — a single residual linear layer). It is not a re-upload of CLAP's weights, and the base CLAP model is unmodified.
• All credit for the CLAP architecture and pretraining belongs to LAION-AI. Use this adapter only together with the original laion/larger_clap_general checkpoint.

What was trained

A single Linear(512, 512) residual adapter (out = normalize(x + W x)) applied to CLAP's frozen audio embeddings, trained with a cross-entropy contrastive objective against CLAP's frozen text-class embeddings.

Training data: ESC-50 folds 1-4 (1,600 clips). Fold 5 (400 clips) was never seen during training and is used exclusively for evaluation below, so these numbers reflect genuine held-out generalization, not memorization of the training set.

Results (ESC-50, held-out fold 5 only)

	Accuracy
CLAP zero-shot (no adapter)	85.25%
+ omura-embed-audio adapter	95.75%

Full reproduction script: benchmarks/eval/clap/finetune_esc50_head.py in the omura-backend repo; see also BENCHMARK_REPRODUCTION.md there.

(For reference, the full-dataset zero-shot number Omura also reports elsewhere — 86.65% — evaluates CLAP across all 2000 ESC-50 clips including the folds used to train this adapter, so it isn't directly comparable to the 85.25%/95.75% held-out figures above; both are honestly reported here for transparency.)

Usage

import torch, torch.nn.functional as F
from transformers import ClapModel, ClapProcessor

base = ClapModel.from_pretrained("laion/larger_clap_general")
processor = ClapProcessor.from_pretrained("laion/larger_clap_general")

ckpt = torch.load("omura_clap_head.pt", map_location="cpu")
head_w = ckpt["state_dict"]  # {"proj.weight": ..., "proj.bias": ...}

def apply_head(x, w, b):
    return F.normalize(x + x @ w.T + b, dim=-1)

# audio_emb = base.get_audio_features(**inputs)  # frozen CLAP embedding
# adapted = apply_head(audio_emb, head_w["proj.weight"], head_w["proj.bias"])

License

Apache 2.0 for this adapter. The base CLAP model is subject to LAION's own license terms.

Attribution

Base model: laion/larger_clap_general by LAION-AI. This repo is a small trained adapter on top of that frozen model — no CLAP weights are included or modified here.