Model card for naflexclap_base_pf8_pt16_moderntextp.160m

Four NaFlexClap checkpoints benchmarked on the standalone CLAP suite + MAEB(beta)-30.

col	config (--model)	checkpoint run	params	samples seen
A	naflexclap_base_pf8_pt16_moderntextp	naflexclap-base-pf8-pt16-mtp_160M_..._seq512_tok131072-90M-0	160.6M	160M
B	naflexclap_base_pf8_pt20_moderntextp	naflexclap-base-pf8-pt20-mtp_240M_..._seq408_tok117504-90M-0	160.6M	240M
C	naflexclap_mediumd_pf4_pt20_moderntextp	naflexclap-mediumd-pf4-pt20-mtp_160M_..._seq816_tok208896-90M-0	139.7M	160M
D	naflexclap_base_pf8_pt16_moderntext	naflexclap-base-pf8-pt16-mt_160M_..._seq512_tok131072-90M-0	159.9M	160M

Methodology

• Standalone CLAP suite: audio uncapped (batch-max padding); classification rows show default prompt ("This is a sound of {c}.") and best-of-sweep; retrieval is A2T/T2A = text-retrieval / audio-retrieval R@5.
• MAEB(beta)-30: --batch_size 16, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, uncapped except BirdCLEF capped at 60 s for comparability (A 0.604 was pre-capped; B/C/D recomputed at 60 s).
• Precision: fp32 weights+activations, no autocast, TF32 matmuls (the MAEB encoder default). Not a precision artifact.

---

1. Architecture, params, geometry

Audio tower = embed/depth/heads; text tower = width/layers/heads.

	A …pf8_pt16_moderntextp	B …pf8_pt20_moderntextp	C …mediumd_pf4_pt20_moderntextp	D …pf8_pt16_moderntext
Audio tower	768/12/12	768/12/12	512/20/8	768/12/12
audio MLP	GELU ×4	SwiGLU ×8/3	GELU ×4	GELU ×4
audio gated attn	no	no	yes	no
audio pooling	MAP	MAP	MAP	MAP
patch geom (pf/pt)	pf8/pt16	pf8/pt20	pf4/pt20	pf8/pt16
freq tokens	8	8	16	8
audio tokens/sec	50	40	80	50
audio tokens @10 s	504	408	816	504
audio params	92.9M	93.0M	72.0M	92.9M
Text tower	512/12/8	512/12/8	512/12/8	512/12/8
text attention	bidirectional	bidirectional	bidirectional	causal
text pooling	MAP	MAP	MAP	EOS
text MLP	SwiGLU	SwiGLU	SwiGLU	SwiGLU
text qk-norm	yes	yes	yes	no
text attn/mlp bias	no	no	no	yes
text gated attn	yes	yes	yes	yes
text params	67.7M	67.7M	67.7M	67.0M
Total params	160.6M	160.6M	139.7M	159.9M
Samples seen	160M	240M	160M	160M

What each comparison isolates:

• A vs D — text tower only (bidir+MAP+qk-norm vs causal+EOS+bias), everything else equal. Cleanest.
• A vs C — audio trunk (768/12 ungated GELU vs 512/20 gated GELU) + geometry (pf8/pt16 vs pf4/pt20).
• A vs B — confounded 3 ways: pt16→pt20, 160M→240M samples, audio GELU→SwiGLU. Not attributable to one.

---

2. Standalone CLAP suite

A2T/T2A = text-retrieval / audio-retrieval R@5.

eval	A	B	C	D
ESC-50 — default prompt	0.9155	0.9270	0.8665	0.8400
ESC-50 — best prompt	0.9155	0.9365	0.9105	0.9060
UrbanSound — zero-shot default ‡	0.7779	0.7177	0.4560	0.4936
UrbanSound — zero-shot best ‡	0.8043	0.7876	0.7759	0.7072
UrbanSound — probe 10-fold (official) ‡	0.8560	0.8414	0.8383	0.8441
AudioCaps A2T/T2A	0.730/0.735	0.749/0.750	0.716/0.721	0.678/0.664
Clotho-joined A2T/T2A	0.584/0.538	0.555/0.645	0.603/0.599	0.502/0.564
Clotho-std A2T/T2A	0.445/0.524	0.553/0.468	0.500/0.444	0.462/0.404
GTZAN — best	0.758	0.769	0.776	0.779
BirdCLEF (60 s cap)	0.604	0.635	0.622	0.577
VGGSound — top-1, best prompt ¶	0.4184	0.4803	0.4772	0.4307
VGGSound — top-5 (default prompt) ¶	0.6928	0.7900	0.7845	0.7518

Classification is prompt-sensitive: default "This is a sound of {c}." is pathological for some checkpoints (D UrbanSound 0.495→0.709 best; C 0.456→0.775 best). Best-prompt is the fairer signal. GTZAN best uses "This is a {c} song."/"a {c} song"; UrbanSound/ESC-50 best vary per model.

‡ UrbanSound rows use the full danavery/urbansound8K (8732 clips), matched data for zero-shot + probe.

Probe = official UrbanSound8K protocol: supervised logreg on audio embeddings, 10-fold CV on the predefined fold splits (train 9 / test 1, ×10, averaged) — literature-comparable. All four land at 0.838–0.856 (1.8 pt spread) despite zero-shot ranging 0.46–0.80, so the entire UrbanSound zero-shot spread is text-alignment / prompt, not audio quality: e.g. D probes 0.8441 (above B and C) yet its default zero-shot was 0.494 (EOS-pooling text misalignment; audio is fine). (A naive random 5-fold inflates the probe to ~0.93 via same-recording leakage — do not use it; the UrbanSound8K authors explicitly warn against reshuffling.)

¶ VGGSound = SINGLE-label, metric is top-1/top-5 accuracy (309 classes; not mAP). Source mteb/VGGSound test (9,888 clips, audio + integer label, ready 309-class ClassLabel vocab — no reconstruction); audio uncapped; VGGSound prefers a template (bare {} worst 0.22–0.26); B/C/D peak at "This is a sound of {c}.", A at "the sound of {c}"; ordering B > C > D > A (A's text tower aligns differently, ~6 pt best-to-best). Harness validated against LAION-CLAP (laion/clap-htsat-unfused through the same pipeline = 0.234 top-1, ≈ its published VGGSound zero-shot ~0.26).

---

3. MAEB(beta)-30 — full per-task (main_score)

task	A	B	C	D
BeijingOpera	0.9110	0.9194	0.8688	0.8773
BirdCLEF (60 s cap)	0.6040	0.6350	0.6220	0.5770
CREMADPairClassification	0.5505	0.5653	0.5507	0.5616
CREMA_D	0.3209	0.3596	0.3140	0.3566
CREMA_DClustering	0.0059	0.0236	0.0088	0.0176
ClothoT2ARetrieval	0.3823	0.3971	0.3748	0.3219
CommonLanguageAgeDetection	0.1695	0.1688	0.1708	0.1641
CommonVoiceMini21T2ARetrieval	0.7396	0.7643	0.7007	0.5387
FSD2019Kaggle	0.5639	0.5532	0.5605	0.5694
FleursT2ARetrieval	0.4673	0.5171	0.4136	0.2682
GTZANAudioReranking	0.8399	0.8366	0.8399	0.8393
GTZANGenre	0.8600	0.8520	0.8490	0.8570
GigaSpeechT2ARetrieval	0.0016	0.0013	0.0006	0.0021
IEMOCAPGender	0.9025	0.8944	0.9057	0.8926
JamAltArtistA2ARetrieval	0.9041	0.8927	0.9011	0.8931
JamAltLyricA2TRetrieval	0.4102	0.4101	0.2920	0.2642
MACST2ARetrieval	0.2952	0.2952	0.2799	0.2748
MInDS14	0.2836	0.3218	0.2707	0.2453
MridinghamTonic	0.3827	0.4632	0.3545	0.3517
NMSQAPairClassification	0.6454	0.6670	0.6735	0.6241
RavdessZeroshot	0.2806	0.2833	0.3090	0.2715
SIBFLEURS	0.1711	0.1697	0.1673	0.1708
SpeechCommandsZeroshotv0.02	0.4840	0.3130	0.4543	0.5231
SpokenSQuADT2ARetrieval	0.2233	0.2500	0.1900	0.1733
UrbanSound8KT2ARetrieval †	0.0088	0.0094	0.0088	0.0090
VehicleSoundClustering	0.0251	0.0047	0.0183	0.0177
VoxCelebSA	0.2844	0.3041	0.2708	0.2679
VoxPopuliAccentPairClassification	0.5598	0.5538	0.5455	0.5580
VoxPopuliGenderClustering	0.3871	0.1977	0.1322	0.3577
VoxPopuliLanguageID	0.9720	0.9760	0.9660	0.9820
MEAN (30)	0.4545	0.4533	0.4338	0.4276

† UrbanSound8KT2ARetrieval ≈ 0.009 is degenerate task data (10 labels, 1-to-1) — not a model signal.

---

4. Audio-representation probes (supervised, prompt-independent)

Two supervised linear probes (logreg) on the audio embeddings — isolating the audio tower from text/prompt alignment, to test whether the wide MAEB / zero-shot spreads reflect real audio-representation differences.

probe — metric	A	B	C	D
VoxPopuli gender — balanced-acc	0.9688	0.9677	0.9669	0.9695
VoxPopuli gender — AUC	0.9876	0.9872	0.9871	0.9881
VoxPopuli gender — macro-F1	0.9581	0.9567	0.9551	0.9580
UrbanSound8K — 10-fold acc (§2)	0.8560	0.8414	0.8383	0.8441

• VoxPopuli gender — facebook/voxpopuli test, all 16 language configs (n = 16,991, 69% male, 1133 speakers). Speaker-disjoint GroupKFold(5) (same speaker ⇒ same gender, else a random split leaks), class_weight=balanced; read straight from the cached parquet (no datasets-arrow rebuild).
• UrbanSound8K — the official 10-fold probe from §2 (full danavery 8732, predefined folds), shown here for the audio-representation view.

All four audio towers encode gender identically — AUC 0.987–0.988 (0.10 pt spread), bacc 0.967–0.970. The three metrics agree and rank the models as a tie (imbalance doesn't pull them apart because separability is so high; AUC = cleanest "is-it-separable" number, raw accuracy would be the trap). This is the second probe (after UrbanSound, 0.838–0.856, 1.8 pt spread) showing the audio towers are equivalent — so the large MAEB spreads on VoxPopuliGenderClustering (0.13–0.39) and UrbanSound8KT2ARetrieval were task/metric artifacts (500-sample clustering noise; degenerate 1-to-1 retrieval), not audio-representation gaps. Note D (causal/EOS) is marginally best on both probes despite its weak MAEB clustering — its weaknesses are entirely text-side. (MAEB's gender task uses mteb/mini-voxpopuli, only 500 samples across 5 langs — hence the noise.)

---

Key findings

1. Text tower (A vs D, clean isolation): bidirectional+MAP > causal+EOS — MAEB 0.4545 vs 0.4276 (−2.7 pt). The loss concentrates on rich-text retrieval (CommonVoice −0.20, Fleurs −0.20, JamAltLyric −0.15, Clotho, SpokenSQuAD, MInDS14): EOS pooling compresses the whole caption into the last token; MAP keeps the full sequence retrieval needs. Even/better on short-label classification (SpeechCommands +0.04, CREMA_D +0.04). Causal+EOS is also more prompt-fragile (its weak default-prompt zero-shot undersells it).

2. Longer training (A vs B, +confounds): helps the CLAP suite, flat on MAEB. B is best on ESC-50/AudioCaps/ Clotho/BirdCLEF but ties A on MAEB (0.4533 vs 0.4545) — two big drops (SpeechCommands −0.17, VoxPopuliGenderClustering −0.19) cancel broad gains; excluding those two, B is +1.2 pt over A on the other 28. B's SpeechCommands −0.17 is checkpoint-specific (D, also causal-free pt16, is +0.04 there).

3. mediumd audio trunk (A vs C): −2.1 pt MAEB, but C is 17% smaller (139.7M) and the gap is task-dependent (wins BirdCLEF/GTZAN, loses speech/temporally-fine). No robust pt16-vs-pt20 effect once prompts are matched.

4. No pt20 UrbanSound weakness and no fill/precision artifacts — UrbanSound clips are ≤4 s (~21 tokens), too short for pt to matter; per-class pt20 gains on jackhammer; pad-mode (floor/silence/repeat) moves SpeechCommands <0.003; MAEB runs fp32.

Result provenance (persistent)

• Eval scripts (under those dirs): clapv2_cls_sweep.py, gtzan_sweep.py, birdclef_capped.py (argv model/ckpt/cap/out), urban_perclass.py, urban_durations.py, speechcmd_padmode.py, urban_full.py (full-8732 danavery: zero-shot sweep + official 10-fold probe in one embedding pass; urban_probe10.py = probe-only; urban_probe.py = old random-5-fold, superseded); vox_gender_probe.py (VoxPopuli gender, speaker-disjoint GroupKFold; VOX_LANGS env selects configs; results in voxgender.log = 5-lang, voxgender16.log = full 16-lang).

---

Appendix — pf8 vs pf4 (frequency tokens): confounded, but a directional signal

Can we isolate a pf effect? Not cleanly. pf4 appears in exactly one checkpoint — C (mediumd_pf4_pt20) — which differs from the base pf8 models by more than pf: a narrower trunk (mediumd 512 vs base 768) and gated audio attention (only C). pt is not a confounder here — B (base_pf8_pt20) vs A (base_pf8_pt16) isolates pt and came out ~neutral — so C-vs-base = pf4 ⊕ trunk ⊕ gating, with trunk + gating the live confounds.

But the pattern isn't random. Against A (the clean pair — pf differs, same 160M-sample training):

task type	task	C (pf4)	A (pf8)	Δ (C−A)
spectral / tonal	BirdCLEF (60 s cap)	0.622	0.604	+0.018
spectral / tonal	GTZAN (best)	0.776	0.758	+0.018
spectral / tonal	VGGSound (top-1, best)	0.4772	0.4184	+0.059
broadband / semantic	ESC-50 (best)	0.9105	0.9155	−0.005
broadband / semantic	AudioCaps (A2T)	0.716	0.730	−0.014
broadband / semantic	FSD50K (bare)	0.3624	0.3895	−0.027
broadband / semantic	MAEB-30 (mean)	0.4338	0.4545	−0.021

Caveats (why "suggestive", not proven): gating is co-located in C and could drive (some of) the spectral wins; and the win costs 2× freq tokens — a resolution/compute trade, not free.

To settle it: one controlled run — base-pf4-pt20 (768 trunk, gating off, same recipe) vs B (base-pf8-pt20) — neutralizes trunk + gating + pt and leaves pf as the only mover. Expected from the above: small gains on spectral / tonal tasks, ~flat-to-slightly-down elsewhere, at 2× the freq-token cost.