p2-flat-d896-93cdf5a9

One of 13 runs from the SODA-extension flat-vs-hier campaign: a compute-matched comparison of the flattened SODA architecture (paper) against a CSM/Moshi-style hierarchical factorization (backbone over steps + small depth transformer over the 8 Mimi RVQ codebook slots), trained on the same interleaved text+audio corpus with a shared id-space so held-out NLL is directly comparable across arms.

• 🌐 Project page: https://soda-audio.github.io
• 💻 Code + full study: https://github.com/potsawee/marin/tree/soda-extension/experiments/audio — results and analysis in report/FINDINGS.md
• 🗂️ All models: soda-research

Model details

Architecture	flattened (single decoder over the flat token stream)
Loss recipe	uniform cross-entropy over all tokens
Compute budget (3× forward FLOPs)	3e18
Model	d=896, L=9
Window	4096 tokens
Total parameters (incl. embeddings)	375M
Training step	48960
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p2-flat-d896-93cdf5a9")
model = AutoModelForCausalLM.from_pretrained("soda-research/p2-flat-d896-93cdf5a9")