gatilin/llama.cpp-youtu-moe

llama.cpp support for Youtu-LLM-A7B-Base

Patch that adds YoutuVLMoEForConditionalGeneration (youtu-moe arch) to llama.cpp — text and vision.

Item	Value
Upstream model	tencent/Youtu-LLM-A7B-Base @ 6b0d02b2912c2975806db5a4922a1e091fa2f5d3
llama.cpp base commit	e0f142d5ddc1461ce721644d92af33db23545105
GGUF weights	gatilin/Youtu-LLM-A7B-Base-GGUF
Status	text + vision verified end-to-end on real 120B weights (macOS / Metal)

中文摘要：本仓库提供一个 llama.cpp 补丁，为 Youtu-LLM-A7B-Base 增加 youtu-moe 架构支持（KDA 线性注意力 + GatedMLA 混合、256 专家 QB 路由 MoE、 Qwen3-VL 视觉塔）。已验证文本与图文推理，可导出 mmproj、可量化。下面第 6 节记录了 几个踩过的坑（RoPE 布局、mrope sections、模态感知专家偏置、</think> 残留）。

---

1. Quick start

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout e0f142d5ddc1461ce721644d92af33db23545105
git apply ../llama.cpp-youtu-moe.patch

cmake -B build -DGGML_METAL=ON -DLLAMA_BUILD_SERVER=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server llama-mtmd-cli llama-quantize

On macOS, cmake / ninja are often only inside a Python venv (pip install cmake ninja).

Text inference:

./build/bin/llama-cli -m youtu-moe-a7b-Q4_K_M.gguf --jinja \
  -p "用一句话解释什么是光合作用。" -st --simple-io -ngl 99 -n 128

Image inference:

./build/bin/llama-mtmd-cli -m youtu-moe-a7b-Q4_K_M.gguf \
  --mmproj youtu-vl-mmproj-f16.gguf --image test.jpeg \
  -p "<|vision_start|><__media__><|vision_end|>描述这张图片的内容。" -n 200 -ngl 99

The <|vision_start|>…<|vision_end|> wrapper is required. chat_template.jinja in the upstream repo has no image branch, so --jinja cannot inject the image placeholder — feed the raw prompt instead. <__media__> is mtmd's media marker; do not also write <|image_pad|>.

2. What's in the patch

File	Change
src/models/youtu-moe.cpp	new — the arch: graph, KDA layer, MLA layer, QB-MoE, modality input
src/models/models.h	new — llama_model_youtu_moe class + graph members
conversion/youtu_moe.py	new — HF → GGUF converter (YoutuVLMoEForConditionalGeneration)
conversion/youtu_vl_vision.py	new — visual tower key mapping for --mmproj
src/llama-arch.cpp / .h	register LLM_ARCH_YOUTU_MOE and its metadata keys
src/llama-model.cpp	register the arch; route it to the IMROPE group
src/llama-graph.cpp	build_inp_mem_hybrid_k() helper (KDA + MLA hybrid KV cache)
src/llama-context.cpp	hybrid recurrent state bookkeeping
gguf-py/gguf/constants.py	tensor/arch/metadata key names
conversion/__init__.py	register the converter module

3. Conversion

# put the 24 shards under models/Youtu-LLM-A7B-Base/ first
bash convert_youtu.sh              # 7 steps, idempotent

convert_youtu.sh runs: weight integrity check → export mmproj → BF16 GGUF → metadata self-check → Q4_K_M quantize → text smoke test → vision smoke test.

Two flags matter:

• --use-temp-file is mandatory for BF16. Without it the converter keeps every tensor in RAM and a 220 GiB model OOMs on a 128 GiB machine.
• Quantize is done from the BF16 GGUF, not from safetensors.

Produced artifacts:

File	Size
youtu-moe-a7b-bf16.gguf	220 GiB (1090 tensors)
youtu-moe-a7b-Q4_K_M.gguf	66 GiB
youtu-vl-mmproj-f16.gguf	867 MiB

4. Architecture notes

Youtu-LLM-A7B: 48 layers, hidden 3072, ~120B total / ~7B active, vocab 129280, context 262144, 262144-token training window over 11.5T multimodal tokens.

Hybrid attention. 3 KDA (linear attention) layers followed by 1 GatedMLA layer, repeated 12 times. KDA reuses llama.cpp's gated-delta-net recurrent path (llm_build_delta_net_base, build_recurrent_attn, conv state + ssm state, L2-normalised q/k, gated lower bound −5.0). MLA is the DeepSeek-style low-rank path with q_lora_rank=1024, kv_lora_rank=512, qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128, plus a head-wise output gate ([32, 3072]) applied after the attention output projection.

MoE. 256 routed experts, top-8, SwiGLU, width 1024, plus 1 shared expert. Routing is quantile-balancing with a modality-aware expert bias (qb_beta, shape [2, 256]: row 0 = text, row 1 = image/video):

selection = router_logits - qb_beta[modality]
top8      = argmax8(selection)
weights   = sigmoid(router_logits[top8]) / (sum + 1e-20) * 2.83

Layer 0 is a dense FFN (12288).

Vision. Qwen3-VL-MoE-style ViT + merger (clip.projector_type=qwen3vl_merger), exported as a separate mmproj.

5. Gotchas that cost real debugging time

5.1 RoPE layout: ggml IMROPE is NEOX, the official model is not

ggml's MROPE / IMROPE force NEOX rotation (first half paired with second half), while the official MLA implementation does cat(q[0::2], q[1::2]) + rotate_half — i.e. adjacent pairs. The two must be reconciled by permuting the weights at conversion time:

# wq_b: per-head [q_nope(128) | q_pe(64)]
q  = w.reshape(32, 128 + 64, -1)
pe = q[:, 128:]
w  = torch.cat([q[:, :128], pe[:, 0::2], pe[:, 1::2]], dim=1).reshape(32 * 192, -1)

# wkv_a_proj_with_mqa: [kv_lora(512) | k_pe(64)]
pe = w[512:]
w  = torch.cat([w[:512], pe[0::2], pe[1::2]], dim=0)

Verified numerically: after the permutation the per-element error vs. the official implementation is 1e-16; without it the relative error is 1.33 (garbage).

5.2 mrope sections must be [11, 10, 10, 0], not [12, 10, 10]

The official config says mrope_section=[12, 10, 10] (sum = 32 = qk_rope_head_dim/2). But ggml computes sector = (freq_index / 2) % sum(sections). With sum = 32, freq 31 lands in the extra slot and reads pos[3] — which is always 0 for text tokens, silently corrupting one frequency. Setting sections[0] = 11 (sum = 31) maps freq 31 → sector 0 → T and leaves every other frequency unchanged, matching the official _apply_interleaved_mrope frequency-by-frequency (verified by exhaustive check).

5.3 Modality routing is decided by "is this row an embedding", not by token id

mtmd injects image embeddings via llama_batch_ext_add_embd, so for a pure-vision batch ubatch->token == nullptr while ubatch->type / embeddings are set. The graph input llm_graph_input_youtu_modality therefore emits 1 for every token of an embedding-only batch, 1 where ubatch->type != 0 otherwise, and 0 for plain text. Two rows of qb_beta differ substantially (mean |Δ| = 0.387, max 2.63, 100 % non-zero), so getting this wrong visibly changes expert selection.

Also: when the last layer trims inpL with inp_out_ids, the modality ids must be trimmed with the same ggml_get_rows(...) call, or the shapes stop matching.

5.4 The model always emits a leading </think>

This is a base model — the upstream README says raw text continuation is the recommended mode and chat_template.jinja is only provided "for further purpose". It nonetheless emits </think> (token 128014) as its very first assistant-side token, closing the think block the template pre-opened:

prompt:  <｜Assistant｜><think>      <- template
model:   </think>光合作用是…          <- generated

It does this even with a bare <｜User｜>…<｜Assistant｜> prompt that contains no think markers at all, and even if the prompt already ends with </think>. </think> is token type USER_DEFINED (4), so llama.cpp prints it as ordinary text.

llama.cpp's auto parser (common/chat-diff-analyzer.cpp) cannot classify it: the official template renders <think> in thinking mode and </think> otherwise, which is a replacement diff, and compare_thinking_enabled() only recognises append-style diffs (its 64-char tail anchor never matches on the other side) → reasoning_mode is judged NONE and the marker stays in message.content.

conversion/youtu_moe.py therefore rewrites the non-thinking branch to emit <think></think> (semantically identical — an empty think block), which turns the diff into an append and makes the analyzer report reasoning_mode: TAG_BASED (-lv 5). The parse itself still does not strip the token in current llama.cpp (p.prefix(generation_prompt, reasoning_start) anchors on a prompt literal that is not present in the generated text), so strip_think.py removes it on the output side.

Tried and not recommended: banning token 128014 via logit_bias. The </think> disappears, but the model then believes it is still inside the think block and starts emitting quasi-reasoning prose — strictly worse.

5.5 Small mtmd-CLI annoyances

• Log timestamps have four dot-separated groups (0.05.132.418 I …); a three-group regex will not filter them.
• llama-mtmd-cli prints the generated text with LOG(), so --log-disable swallows the answer too (symptom: empty output).

6. Known limitations

• No reasoning extraction into reasoning_content (see 5.4) — use strip_think.py.
• Only Q4_K_M and BF16 are published; other quant types are untested.
• KDA recurrent path has only been validated on Metal (macOS). CUDA/CPU are expected to work but have not been run.
• Small-text OCR is limited by the default 300 image tokens; pass --image-min-tokens 1024 for dense documents.

7. Verified output

Text (Q4_K_M, M3 Max 128 GiB, prefill ≈ 38 t/s, decode ≈ 14 t/s):

> 用一句话解释什么是光合作用。
光合作用是植物、藻类和某些细菌利用光能将二氧化碳和水转化为有机物（如葡萄糖）并释放氧气的过程。

Vision — the 1969-07-21 New York Times front page (mtmd's bundled test-1.jpeg): masthead, date MONDAY, JULY 21, 1969, LATE CITY EDITION, 10 CENTS, VOL. CXVIII. NO. 46,731, headline MEN WALK ON MOON, subhead ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG — all correct.

8. License

The patch follows llama.cpp's MIT license. The weights and the model architecture follow the upstream Youtu-LLM license (license: other); see tencent/Youtu-LLM-A7B-Base/LICENSE.txt.