Gemma 4 12B Unified: encoder-free audio comprehension degrades deterministically past ~450–500 tokens of absolute position

TL;DR. On gemma4_unified (the encoder-free multimodal 12B), audio placed beyond roughly 450–500 tokens of absolute position in the sequence is no longer understood — deterministically, at temperature 0. Three controls isolate the cause to the encoder-free audio path itself: (a) vision on the same checkpoint survives at ~800 tokens (OCR included), (b) E2B and E4B (USM-Conformer audio encoder) survive at every position we tested in the same runtime, and (c) giving audio tokens intra-block bidirectional attention (off-spec) largely restores comprehension. Our working hypothesis: encoder-free audio tokens (raw 640-sample frames through a linear projection) build their representations via causal attention inside the LLM, and those representations are fragile to long prefixes never seen in training; vision is robust because its tokens get intra-block bidirectional attention. The practical fallout: direct-audio multi-turn chat is unreliable — any prior turn (even text-only) pushes the current clip past the cliff.

We would love to know whether this matches the training distribution of the unified audio path (audio always near the start of the sequence?), and whether there is a recommended maximum prefix length for audio inputs that could be documented.

Setup

The audio is attached to the first user turn together with a short instruction ("The person said this to you by voice. Reply to what they said."). The absolute position of the audio block is pushed by growing the system prompt: a fixed ~250-token persona followed by semantically null filler (repeated "- "), so no distracting content is added — position is the only variable. With template overhead, the audio sits at roughly 320 absolute tokens in the control case. Fresh session (empty KV cache) for every case.

Finding 1: positional cliff, audio only

12B Unified, null filler (does the reply engage with "seventeen"?):

The cliff sits somewhere around 450–500 absolute tokens. With semantic filler (real sentences) instead of null filler it arrives slightly earlier (fails from ~+120), but the null-filler runs show the effect needs no distracting content: it is positional, not attentional-competition.

Controls, same protocol:

Practical consequence (multi-turn): with the same clip and prompt, A) fresh session → ✅ "Ah, el diecisiete…"; B) one text-only prior turn in context → ❌ "No tengo ni idea de qué estás hablando. ¿Qué es eso del 17?" (echoes a fragment, no comprehension of the utterance); C) one prior audio turn → ❌. Any conversational prefix is enough to cross the cliff, so encoder-free direct-audio chat only works reliably on the first turn of a session.

Finding 2 (secondary): extreme sensitivity to the clip tail

Same 12B checkpoint, transcription prompt, temperature 0, one-shot:

The zero-pad doesn't just prevent the failure, it rescues already-truncated clips. Gain is irrelevant (raw / peak-normalized / 4× / 10× all behave identically once the tail is padded). Consistent with the encoder-free ingestion: an abrupt ending or a partially-filled final 640-sample frame corrupts the last-frame feature exactly where the closing word lives. Easy workaround, but worth documenting; a Conformer front-end wouldn't care.

What we ruled out (implementation audit)

Since the reproduction runs on our Swift/MLX port, we audited it line-by-line against two references before concluding anything about the model:

• Attention mask: in both HF transformers (modular_gemma4.py) and google-deepmind/gemma (gemma4/_transformer.py), audio tokens get causal attention only — the intra-block bidirectional override applies to vision types only. Our port matches.
• Positions: uniform arange + past_seen_tokens, no audio-specific position handling anywhere in the references. Our port matches.
• RoPE: partial_rotary_factor=0.25 with the "proportional" parameterization (modeling_rope_utils.py) — frequencies computed over the full head_dim and the (i, i+half) pairing — verified identical. Attention scale 1.0 verified. norm→rope order verified.
• Placeholder scatter: token-id–based with a count guard; the vision control (which shares mask/RoPE/scale/scatter with audio) surviving at ~800 tokens is the strongest evidence the shared stack is healthy.
• Discriminating experiment: patching the mask (off-spec) so audio blocks also get intra-block bidirectional attention largely restores positional comprehension (at +240/+480 the model at least understands "a number was said" where causal understood nothing) — but destabilizes the chat template (control-token leakage), as expected for an out-of-training-distribution mask. Not a fix; evidence for the mechanism.

Honest caveats

• The reproduction stack is Swift/MLX. To our knowledge there is no runnable Python reference for 12B unified audio to cross-check against (mlx-vlm implements Gemma 4 audio only for the 2B/4B Conformer models; running the 12B in PyTorch transformers was impractical locally). A residual possibility remains that a positional-extrapolation bug is shared with our reading of the references — but the vision control and the E2B/E4B controls run through the very same code paths in our stack, and they don't degrade.
• All checkpoints are 4-bit quantized (mlx-community conversions). We consider it unlikely that quantization produces a clean position threshold that spares vision, but we haven't run an fp16 control.
• Clips are macOS TTS (voice "Paulina", es-CL Spanish). The effect reproduces with different sentences and clip lengths; we have not swept languages.

Reproduction

1. Fork + branch above (git checkout 47a2432), or any runtime that can feed gemma4_unified audio.
2. Generate the clip with the say command in Setup; load at 16 kHz mono float32; append 0.25 s of zeros, round length up to a multiple of 640 samples.
3. System prompt = any ~250-token persona + String(repeating: "- ", count: N) for N ∈ {0, 80, 240, 480}.
4. First user turn: text "The person said this to you by voice. Reply to what they said." + the audio clip. Temperature 0, fresh session per case.
5. Score: does the reply engage with the fact in the clip ("seventeen")?
6. Vision control: same protocol, replace the clip with any image containing shapes + rendered text; ask for a full description.
7. E2B/E4B control: same protocol, swap the checkpoint.

Happy to share the full experiment harness (a small Swift CLI with all the modes used here: position, imgposition, context, tail, prompts) and raw transcripts if useful.

Related but separate issue we hit along the way: generation_config.json's suppress_tokens lists only 258882/258883 — the rest of the multimodal placeholder family (image/video soft tokens, boi/eoi/boa/eoa markers) is not listed, and runtimes that don't implement suppress_tokens sample even the listed ones into text-only replies (reported independently in jundot/omlx#1670; we hit it too and fixed it runtime-side). Might be worth completing the list in the shipped generation_config.json.