timiiowolabi/muta-compress-20260920

Muta Compress — 2026-09-20 refinement campaign

Research artifact, not a finished product. This repository holds the checkpoints and GGUFs produced by a compression-and-refinement campaign on the Muta STEM tutor (originally a LoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct). It is honestly a mixed result: a real, measured speed/memory gain and capability numbers that trend the right way but aren't individually significant, judged answer quality that is level with (not better than) the model it replaces, two missed targets, and one clean negative result (a tutoring-scaffolding probe). Read the whole card before trusting a number.

Full write-up, every raw measurement, and the evidence trail: RESULTS.md's 2026-09-21 entry and docs/compression-pipeline-results.md in the Muta repository (bench/measurements/refine-20260920/).

Current best: refine-final/gguf/refine-qat100-Q4_0.gguf

A dense 26-layer Qwen2 model, width-pruned (FFN 8960 → 7168 by measured neuron importance), verified-distilled from a Qwen2.5-7B-Instruct teacher, then quantization-aware trained toward pure Q4_0 for 100 of a planned 362 steps (a GPU-box outage stranded the rest — see "Known limitations" below).

Property	Value
Parameters	1,051,201,024 (1.05 B)
Layers	26
Hidden size	1536, FFN 7168 (pruned from 8960), vocab 32,000, tied embeddings
Quantization	pure Q4_0
File size	592,887,552 bytes
sha256	6e10045045eada66ba21533b987a725d1c7ab573adcd26024ecd3c078a04d3fc
llama.cpp build	b10175, scalar/SSSE3 CPU kernels

Measured, on the scalar llama.cpp b10175 audit build, 4 threads, GCP audit boxes (raw evidence: bench/measurements/refine-20260920/final/):

Metric	Value	vs. dense parent (s2-26L-Q4_0)
ARC-Easy-50 (audit set)	72	70
ARC-Easy-500 (500 held-out items)	73.4 %	70.0 %
GSM8K-100 (greedy, exact final-number match)	53 %	49 %
Decode speed	15.51 tok/s	13.15 tok/s
Peak RSS	705.7 MB	809.9 MB
Blind-judge dev S_total (30 held-out prompts, campaign scoring)	73.86 / 74.45 (two independent judge batches)	69.54 / 69.21 (same batches, same control)
Judges' acc, dev (30 prompts, raw, 0–10 scale ÷10)	31.3 / 33.7	32.0 / 30.7 (same two batches)
Judges' acc, official ten (n = 10, noisy)	35.0 / 32.0	41.0 / 45.0 (same two batches)

Read the raw Judges' rows above before crediting this model with "better answers." This is the campaign's row of record on S_total — +4.3 to +5.2 points over the dense parent, graded twice by independent judge batches to guard against grader-batch noise (both landed within 0.6 points of each other) — but the gain is not driven by judged answer quality. The same dense-parent answers have been graded blind four times across this campaign's chain (dev: 28.0, 31.0, 32.0, 30.7; official: 45.0, 44.0, 41.0, 45.0) — a ≈4-point dev spread for identical answers, this rubric's own same-batch grader-noise floor. This model's own Judges' scores (31.3, 33.7 dev; 35.0, 32.0 official) sit inside, or below, that same spread: judged quality is level with the parent (dev) or worse (official ten). Decomposed into the scoring formula's own components, only ≈0.6–1.5 of the ≈4.3–5.2-point S_total gain comes from accuracy/efficiency; ≈3.7 points — the majority — is the pre-registered effect of crossing the 15 tok/s speed cap alone (the spec's own "+3.7" estimate, made before any training ran). The three capability numbers in the table above (ARC-Easy-50/500, GSM8K-100) do move in this model's favour, but at n = 50/500/100 none individually clears conventional statistical significance on its own (ARC-Easy-500: z ≈ 1.2; GSM8K-100: z ≈ 0.6) — three consistent directions, not proof of a capability win. In short: this release is faster and smaller with capability numbers that trend the right way but aren't individually significant, and with judged answer quality that is not measurably better than (and on the noisy official ten, is sometimes worse than) the model it replaces.

Two other exports exist in refine-final/gguf/ for comparison: refine-distilled-Q4_0.gguf (the same checkpoint before QAT) and refine-qat100-R1i-Q4_0.gguf (the QAT checkpoint with an imatrix pass added — a same-batch loss on the held-out dev prompts but a gain on the small, noisy 10-prompt official set; inconclusive, not adopted as the current best).

How to run it (llama.cpp)

# One-shot, single turn, no system prompt:
llama-cli -m refine-qat100-Q4_0.gguf --single-turn -p "A train travels 60 miles in 1.5 hours. What is its average speed?" -n 512

# Server, OpenAI-compatible endpoint:
llama-server -m refine-qat100-Q4_0.gguf -c 4096 -t 4

The GGUF carries its own ChatML chat template (<|im_start|>role\n...<|im_end|>); any llama.cpp front end that reads embedded templates (llama-cli --single-turn, llama-server with a client that requests /v1/chat/completions) will render prompts correctly without extra flags. Recommended: -t 4 on a 4-core budget machine; the audit numbers above were measured at exactly that thread count.

The mode switch — read this before judging tutoring behaviour

Whether you supply a system prompt changes what this model does, by design (and by training data):

• No system prompt (or any system prompt other than the one below) → the model gives a complete, worked answer — the behaviour trained into the large majority of the corpus (single-turn verified math/science rows carry no system prompt at all).

• The exact tutor system prompt below → the model is trained to attempt a scaffolding mode — guide the student step by step, ask before telling, rather than handing over the answer immediately:

  You are Muta, a patient tutor for mathematics and science. Work step by step, show each calculation, keep explanations clear and concise, state the final answer plainly, and stop when the question is answered. Guide the student step by step; ask before telling.

This campaign's own probe shows the scaffolding mode does NOT work yet. Fifty blind-graded MathDial-test dialogues (first tutor turn, tutor system prompt) scored the same across every checkpoint in this chain — dense parent 3.24/10, refine-qat100 3.26/10, refine-qat100-R1i 3.24/10 — and all three models "tell the answer" and/or "state a falsehood" on 33–40 of 50 turns. In practice, with the tutor system prompt, expect the model to lecture through a re-derivation of the answer, not to hold back and ask questions. The working hypothesis (untested) is that the dialogue training rows were distilled from a 7B teacher whose own instinct is to explain everything, at only a 0.6 sampling weight (~9 % of training tokens) — not enough to overrule that prior. If you need reliable scaffolding behaviour, do not rely on this system prompt working as intended; treat the "no system prompt" complete-answer mode as the one behaviour this release actually delivers.

Known limitations

• Arithmetic is still wrong roughly half the time on GSM8K-style problems (53 % exact-match on 100 held-out grade-school math word problems). This is an improvement over the dense parent (49 %), not a solved problem.
• All accuracy numbers involving "judges'" scores come from AI graders (multiple blind LLM graders per batch, shuffled labels, fixed rubric), not humans. The 10-prompt "official" set is small and noisy; treat it as a secondary signal, not the number of record.
• This is a research artifact from a time-boxed campaign, not a finished, validated product. QAT ran 100 of a planned 362 steps (a GPU-box outage stranded the rest, permanently — the lost training state cannot be recovered). The stage-1 termination-gate pass rate (0.900) is not backed by a file in this repository; it is a value the campaign's own controller read off the now-unreachable training box before the outage.
• The tutoring-scaffolding mode does not work — see "The mode switch" above. Do not present this model as an adaptive Socratic tutor without further work.
• ARC-Easy-50 carries roughly ±7 points of binomial sampling noise at n = 50 — treat the 500 item ARC-Easy-500 numbers as the more reliable read of that capability.
• Imatrix quantization (the R1i variant) is inconclusive on the evidence gathered — it helps on one judge batch and hurts on the other; no imatrix variant is the current best.
• Judged answer quality did NOT improve over the dense parent it replaces. This model's own Judges' scores (31.3/33.7 dev, 35.0/32.0 official) are within this rubric's own same-batch grader noise of the parent's (32.0/30.7 dev, 41.0/45.0 official) — level at best, and lower on the noisy official ten. The S_total gain reported above is driven mostly by decode speed (crossing the 15 tok/s cap contributes ≈3.7 of the ≈4.3–5.2-point gain), not by better answers. Do not market this release on judged answer quality.
• A post-hoc code review found real training-data defects after this model was already delivered (full detail in the repository's RESULTS.md, "Post-hoc code review"): a verifier false-"correct" path for math answers, a breadth-data gold rule that could accept a number already given in the question, three MCQ false-"correct" paths, breadth questions that were never checked against benchmark test sets, and one GSM8K test item that leaked into training via a MathDial dialogue. Measured, not estimated: re-checking the verifier against the actual 54,896 trained-on rows flipped 0.40 % of verdicts (222 rows), of which an estimated ≈0.033 % (≈18 rows) were genuinely wrong answers the model trained on; a worst-case contamination check found 2 of 100 GSM8K-100 and 1 of 500 ARC-Easy-500 evaluation items are near-duplicates of training pool questions, which moves neither model's reported accuracy by more than 0.1 percentage points and leaves the reported gap between them unaffected. The ≈25,475 orca breadth rows could not be independently re-measured (raw generations exist only on the unreachable GPU box); the original build-time hand-check bounds their error rate at ≤ ≈10 % at 95 % confidence, no tighter. No number in this card changed as a result of this review, the code was fixed for future use, and this checkpoint was neither regenerated nor retrained — the numbers above are exactly what this model, warts included, was trained on and measured against.

Repository contents

Path	Contents
step2-26L-dense/	An earlier chain step's dense 26-layer checkpoint (bf16) + Q4_0 GGUF — the direct ancestor of this campaign's starting point, superseded by refine-final/ below.
step3-26L-moe-b/	A Mixture-of-Experts conversion of the 26-layer model — scored below the dense 26-layer model on every axis but ARC-Easy; a recorded negative result, not recommended for use.
step5-qat-moe-b/	A quantization-aware-training pass on the MoE checkpoint above — trained but never scored (superseded before evaluation); kept for provenance only.
refine-final/gguf/	The current release. refine-qat100-Q4_0.gguf (recommended, see above), refine-distilled-Q4_0.gguf (pre-QAT), refine-qat100-R1i-Q4_0.gguf (QAT + imatrix, inconclusive).
refine-final/hf-distilled/	The pre-QAT checkpoint in Hugging Face transformers format (bf16), for anyone who wants to continue training or requantize with a different recipe.
refine-final/hf-qat100/	The QAT-step-100 checkpoint in transformers format (bf16 master weights under fake-quant simulation), the direct source of the recommended GGUF above.
refine-final/gguf-other/	Intermediate f16 GGUF exports used to build the Q4_0/R1i files (kept for reproducibility, not recommended for direct use — much larger, no accuracy benefit demonstrated over the Q4_0 export at this model size).
this file	Model card.

The training corpus (verified single-turn rows, breadth top-up, regenerated misses, tutoring dialogues) is not published in this repository or its companion dataset repo (timiiowolabi/muta-distillation-data, which still holds only an earlier v1 snapshot) — the newly built rows for this campaign exist only on a GPU box that became unreachable before they could be backed up. See RESULTS.md's 2026-09-21 entry for the full incident record.

License and data attribution

The model weights and code in this repository are released under Apache-2.0.

Training used teacher-generated completions verified against, or dialogues drawn from, the following datasets. Attribution recorded per source (the stricter of two stated licenses is used where a dataset states more than one):

Dataset	License	Role
GSM8K	MIT	Gold answers for verification; grade-school math word problems
ARC (AI2 Reasoning Challenge, Easy + Challenge)	CC-BY-SA-4.0	Gold answers for verification; multiple-choice science questions
QASC	CC-BY-4.0	Gold answers for verification; multi-hop science QA
OpenR1-Math-220k	Apache-2.0	Gold-verified competition mathematics
orca-math-word-problems-200k	MIT	Breadth generation source (gold-checked final answers)
OpenBookQA	undeclared on its Hugging Face card (unknown); the AllenAI GitHub repository is Apache-2.0	Breadth generation source (4,275 gold-checked rows)
MathDial	CC-BY-SA-4.0 (recorded as the stricter of its two stated licenses)	Tutoring-dialogue training rows (train split only; test split held out, never trained on)
ConvoLearn	MIT	Tutoring-dialogue training rows (top-rated half only)

No weights or completions from held-out evaluation material (the 10 official judge prompts, the 30 dev-judge prompts, ARC-Easy/GSM8K test items, the MathDial test split, TutorMoments) were ever used in training.