You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Sori-1B redistributes NVIDIA's frozen Audio Flamingo Next audio encoder, released under the NVIDIA OneWay Noncommercial License (a copy is in LICENSE). Section 3.3 of that license limits the work and any derivative of it to non-commercial use, and defines non-commercial as academic purposes only. Those terms carry over here in full. Access is granted on these conditions: you use the model for academic research only; you do not redistribute the weights or any part of this repository to third parties; you obtain the author's agreement before releasing a derivative model trained or distilled from it; and you cite this work in any publication whose results depend on it.

Sori-1B: An Audio-Grounded Language Model with an Auditory-Ontology Vocabulary

Language learned only by listening.

_{Every text token is grounded in audio and never learned from text alone, over the Sori Ontology Tokenizer, a vocabulary built as an
ontology of auditory concepts.}

Seoul National University · Human Interface Lab

_{Seonuk Kim · Human Interface Lab · Seoul National University · 🇰🇷}

_{Released 28 August 2026}

Introduction

Audio-language models are usually built by attaching an audio encoder to a large language model that was pretrained, and often instruction-tuned, on text alone. That backbone arrives with a strong prior over what sentences are plausible — and on multiple-choice audio benchmarks a plausible sentence is frequently the correct one, whether or not the model listened. The result is a system that scores well while grounding weakly in the audio.

The effect is easy to measure. We scored NVIDIA's Audio Flamingo 3 on MMAU test-mini under one harness and one prompt, changing nothing but the audio itself.

condition	accuracy	margin over chance	margin retained
audio as given	0.758	+0.503	100 %
audio replaced by silence	0.630	+0.375	74.5 %
chance	0.255	—	—

Raw accuracy is the wrong thing to read here, because a quarter of it is free. What matters is the margin the model wins above chance — and three quarters of that margin is still there when there is nothing to hear. Whatever the remaining quarter is doing, most of the answer was already available in the question and the choices.

The same fragility appears from the other direction. Transforms that leave a clip perceptually intact — loudness normalisation, a small gain change, light denoising, trimming leading silence — still flip 4–6 % of answers, and across the fifty-two transforms we tried the median was about one answer in ten. A prediction that moves when the audio is untouched, and holds when the audio is gone, is not a prediction about sound.

None of this is a fault of any one model. It follows from where the language came from: a decoder that already knows how sentences go will use that knowledge first, and audio becomes a weak correction to a strong prior rather than the evidence the answer rests on.

The goal is not a model that ignores text. A question has to be read and choices have to be read, and language is what makes an audio question answerable at all. The goal is an ordering: audio is the evidence, and text is what turns it into an answer. A model that can answer without listening has the ordering backwards, and so does one that cannot read the question.

Sori (소리, Korean for sound) asks what happens if the decoder never learns language that way. It is trained from scratch, with no language-only pretraining and no pretrained-LM initialisation, and every text token it has ever predicted was predicted in the presence of audio.

The question this project exists to answer is whether that is enough — whether a decoder that has only ever read in the company of sound can reach useful audio understanding, and how much grounding it buys. It is an open question, and this repository is where it is being answered rather than a claim that it already has been.

Parameters

component	parameters	share	trained here
NVIDIA Audio Flamingo Next audio encoder	636,968,960	61.5 %	frozen
decoder	317,955,850	30.7 %	✓ from scratch
token embeddings	37,912,320	3.7 %	✓ from scratch
output head	37,912,320	3.7 %	✓ from scratch
audio projector	4,915,234	0.5 %	✓ from scratch
other	6,400	<0.01 %	✓ from scratch
total	1,035,671,084
trained here	398,702,124	38.5 %

The encoder is NVIDIA's Audio Flamingo Next tower, reused unchanged and never updated: it supplies the acoustic representation and nothing else. Everything above it — projector, decoder, vocabulary and output head — starts from random initialisation and is learned here, and every gradient that shaped it came from predicting text in the presence of audio.

The ratio is the point. Under a third of the model is language machinery, and none of that third was ever fitted to text on its own.

Training

resource	scale
hardware	3 × NVIDIA RTX 4090 (24 GB)
audio	7,412 h · 3.28 M distinct clips
examples	4.75 M
text	82.5 M target tokens

Every target token was predicted with audio in the context. There is no text-only corpus in the schedule.

Usage

Currently English only.

import soundfile as sf
from transformers import AutoProcessor, AutoModelForCausalLM

proc  = AutoProcessor.from_pretrained("snkii/Sori-1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("snkii/Sori-1B", trust_remote_code=True).eval()

wav, sr = sf.read("clip.wav", dtype="float32")   # 16 kHz
if wav.ndim > 1:
    wav = wav.mean(1)                            # mono

generate returns only the newly generated tokens, so the decode needs no slicing.

mcq — the model generates the chosen choice string

choices = ["string", "reed", "organ", "bass"]
inputs  = proc(audio=wav, mode="mcq",
               question="Which instrument is playing?",
               choices=choices,
               return_tensors="pt")
out  = model.generate(**inputs, max_new_tokens=24, do_sample=False)
text = proc.decode(out[0], skip_special_tokens=True)

# the prediction is the choice the generation matches
import re, unicodedata
norm = lambda s: re.sub(r"\s+", " ", unicodedata.normalize("NFKC", str(s)).lower().strip()).strip(" .!?,;:'\"()[]{}")
hit  = [i for i, c in enumerate(choices) if norm(c) == norm(text)]
pred = hit[0] if len(hit) == 1 else -1           # -1: matched no choice

This snippet requires the generation to equal a choice exactly. eval_mmau_test_mini.py instead applies MMAU's own string_match (every token of the answer present, no token unique to a wrong choice), which is more lenient, so the two can disagree on the same generation. Use the benchmark's rule when comparing against benchmark numbers.

qa — an open question, a free-text answer

inputs = proc(audio=wav, mode="qa",
              question="What is the speaker doing?",
              return_tensors="pt")
out  = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(proc.decode(out[0], skip_special_tokens=True))

QA carries no choices, so nothing is matched against: the answer is whatever the model generates. Its attention is the MCQ layout minus the choices — the question steers how the clip is read and never reaches the generator, so the answer has to come out of the audio.

caption

inputs = proc(audio=wav, mode="caption", return_tensors="pt")
out    = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(proc.decode(out[0], skip_special_tokens=True))

asr

inputs = proc(audio=wav, mode="asr", return_tensors="pt")
out    = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(proc.decode(out[0], skip_special_tokens=True))

batches

proc(...) builds a single example. For a batch, build each one and collate them:

items = [(wav_a, "Which instrument is playing?", ["string", "reed", "organ", "bass"]),
         (wav_b, "What is the speaker doing?",   ["singing", "laughing", "coughing"])]

exs   = [proc.build([w], "mcq", target=None, question=q, choices=c, permute=False) for w, q, c in items]
batch = proc.collate(exs)
out   = model.generate(**batch, max_new_tokens=24, do_sample=False)
texts = [proc.decode(o, skip_special_tokens=True) for o in out]

collate pads on the right. Batched and one-at-a-time generation give the same strings.

inference endpoint

handler.py in this repo implements EndpointHandler.

{"inputs": {"audio": "<base64 wav/flac/mp3>",   # or a path / url
            "mode": "mcq",                      # mcq | qa | caption | asr
            "question": "Which instrument is playing?",
            "choices": ["string", "reed", "organ", "bass"]},
 "parameters": {"max_new_tokens": 24}}

[{"generated_text": "bass", "choice": "bass", "choice_index": 3}]

A raw audio body is also accepted and treated as mode="caption".

MMAU test-mini

eval_mmau_test_mini.py scores the model on MMAU test-mini, downloading the benchmark from its official sources: the questions from the official repository (mmau-test-mini.json, 1,000 items) and the audio from the Google Drive archive that repository links, via gdown.

pip install gdown soundfile librosa
python eval_mmau_test_mini.py --model snkii/Sori-1B

It writes mmau-test-mini.sori.json — the official benchmark file with model_output added to every record, plus model_choice_by_likelihood unless --no-score-choices is passed. The official scorer reads model_output and ignores the rest:

python evaluation.py --input mmau-test-mini.sori.json     # or pass --run-official-scorer

It reports two readings. Generation + string match is the primary one and the one the official scorer reproduces. Per-choice likelihood is a diagnostic — it asks which choice the model prefers without requiring it to reproduce the string verbatim, so the gap between the two measures reconstruction rather than selection. Pass --no-score-choices to skip it.

The script also prints the accuracies itself, using the official string_match rule (every token of the answer present in the prediction, no token unique to a wrong choice). That reimplementation agrees with the official scorer.

Sori Ontology Tokenizer

The vocabulary is not a word list learned from text statistics. It is built from an ontology of auditory concepts — a graph in which every concept has a place, and the place has a path up to one of eight top-level axes:

axis	what it names
source	what made the sound — creatures, things, nature, people
music	sound organised as music — genre, instrument, key, tempo, dynamics, technique, mood
speech	sound as speech — language, prosody, voice quality, speaker traits, delivery
scene	where the sound is — acoustic environment, room, place, background
property	how it sounds in itself — loudness, pitch, timbre, texture, duration
time	when and in what order — onset, sequence, repetition, rhythm, change
signal	what happened to the signal — channel, reverberation, coding, distortion, noise
quantity	units and counting — Hz, dB, BPM, ordinals, cardinals

The point is that the units of reading and generation should be the units the model is meant to hear in. A vocabulary inherited from text makes audio concepts arrive as fragments of English; one built from an audio ontology makes them arrive whole, and gives a concept the model has rarely seen a neighbourhood it can reason from.

License

This repository redistributes NVIDIA's Audio Flamingo Next audio encoder unchanged and frozen. NVIDIA released it under the NVIDIA OneWay Noncommercial License, whose Section 3.3 limits the work and any derivative of it to non-commercial use and defines non-commercial as academic purposes only. Section 3.1 requires anyone redistributing it to ship a complete copy of that licence, so the full text is in LICENSE.

Section 3.2 permits separate terms for the parts trained here, provided those terms keep the Section 3.3 limitation and say what they cover. They do, and they cover the decoder, projector, vocabulary, output head and code — not NVIDIA's encoder:

Academic use only. Any use outside academic research is outside both NVIDIA's terms and these.
No redistribution of the weights or of this repository.
No derivative model release without the author's agreement.
Cite the work in publications whose results depend on it.

Access is granted manually, so the files themselves — including the full licence text — are visible only after a request is approved. The terms above are stated here in full so that they can be read before requesting access.

The license field is other rather than a Creative Commons one on purpose: a CC licence grants permissions this project has no authority to grant over NVIDIA's weights.

Citation

@software{kim_sori_1b_2026,
  author  = {Kim, Seonuk},
  title   = {{Sori-1B: An Audio-Grounded Language Model with an Auditory-Ontology Vocabulary}},
  year    = {2026},
  month   = aug,
  date    = {2026-08-28},
  version = {1.0},
  url     = {https://huggingface.co/snkii/Sori-1B}
}

CITATION.cff in this repository carries the same metadata in CFF 1.2.0 form.

Downloads last month: 10

Safetensors

Model size

1B params

Tensor type

I64

F32

BOOL

Inference Providers NEW

Audio-Text-to-Text

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for snkii/Sori-1B

Base model

nvidia/audio-flamingo-next-hf

Finetuned

(1)

this model