VoiceCLAP-Commercial (small)

A fully commercially-licensable voice-text contrastive (CLAP-style) embedding model for speech-emotion and talking-style retrieval. Same 110M dual-tower architecture as laion/voiceclap-small, but trained only on data that permits commercial use — and, perhaps surprisingly, it matches or beats the non-commercial-data model on 4 of 5 benchmarks.

Why this model exists

The standard VoiceCLAP training mix contains CC BY-NC (non-commercial) corpora: Expresso and EARS (both Meta, CC BY-NC 4.0), and the bulk of Emilia (the original 101k-hour split is CC BY-NC 4.0). That makes models trained on the full mix unusable for commercial purposes.

This model removes every non-commercial source and keeps only commercially usable data. A controlled ablation (below) shows the small model loses nothing by doing so.

Training data (all commercially licensable)

Corpus	License	Role
Emilia-YODAS (the CC BY 4.0 subset of Emilia, ~19% of emolia-balanced)	CC BY 4.0	emotion / voice captions
LAION's Got Talent	LAION-released	talking-style captions
Majestrino	in-house	voice captions

The non-commercial Emilia clips are filtered out at training time by clip-id (original-Emilia ids look like EN_B00087_S08178_W000004; the retained YODAS clips carry YouTube-style ids). Expresso and EARS are not used. Captions use the __moss_short__ scheme: the MOSS-Audio-8B-Thinking emotion sentence plus one sampled talking-style sentence, 50/50 mixed with the corpus's own caption.

Architecture

Identical to voiceclap-small — a dual-tower CLAP:

Audio encoder	BUD-E-Whisper-Small: 12 layers × 768 dim × 12 heads, 80-mel @ 16 kHz
Text encoder	all-MiniLM-L6-v2: 6 layers × 384 dim, mean-pooled
Joint embedding	768-d, L2-normalised
Loss	SigLIP sigmoid contrastive + Prototypical Contrastive (PCL, w=0.2)
Total parameters	~110 M

PCL adds 39 learned emotion prototypes and a cross-entropy term on z-scored pseudo-labels derived from emolia's emotion-annotation scalars — a small auxiliary loss that sharpens the emotion subspace.

Results — commercial data costs nothing

Controlled ablation, all arms trained identically (1 node, __moss_short__ + PCL w=0.2, 15 epochs), evaluated at each arm's best epoch:

Training data	emonet top1	emonet ρ	VoiceNet-Emo bal@pp	emolia ρ	MAEB-voice
full mix incl. non-commercial Emilia	0.0712	0.2308	0.6112	0.2019	0.3472
this model — commercial only	0.0721	0.2061	0.6227	0.2034	0.3564

This model wins 4 of 5 metrics — emonet top-1, VoiceNet-Emo balanced accuracy (+0.0115), emolia Spearman ρ, and MAEB-voice (+0.009). It trails only on emonet ρ (−0.025, fine-grained intensity ranking on synthetic audio, where the dropped Emilia diversity helped). On the in-domain emolia benchmark and the 8-task MAEB-voice suite it is strictly better than the non-commercial-data model.

Two recovery experiments were tried and discarded: adding VoxCeleb1/2 (CC BY) hurt, and upweighting the existing safe corpora was flat. Plain commercial data is best.

Absolute numbers are from a 1-node training scale used for the controlled ablation; treat them as relative (commercial-vs-noncommercial deltas), not as the maximum achievable with a full multi-node run.

Usage

import torch, soundfile as sf
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("gijs/voiceclap-commercial", trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained("gijs/voiceclap-commercial")

# audio: raw mono waveform @ 16 kHz
wav, sr = sf.read("clip.wav", dtype="float32")
audio_emb = model.encode_waveform(torch.from_numpy(wav))

# text
t = tok(["a person speaking with quiet pride in their voice"], padding=True, return_tensors="pt")
text_emb = model.encode_text(t["input_ids"], attention_mask=t["attention_mask"])

score = (audio_emb @ text_emb.T).item()  # cosine similarity (both L2-normalised)

Conversion from the training checkpoint was verified functionally against the original open_clip implementation (cosine ≥ 0.99999 on both towers).

License

CC-BY-4.0 — all training data is commercially usable (Emilia-YODAS CC BY 4.0, LAION's Got Talent, in-house Majestrino), and the architecture/weights carry no non-commercial restriction. This is the distinguishing feature of this model versus the standard VoiceCLAP releases, which inherit a non-commercial restriction from their CC BY-NC training data.