midium-ai/KnowLine-26B-A4B-Gen2-8bit

KnowLine-26B-A4B-Gen2 — MLX 8-bit

An 8-bit MLX quantization of PelaAI/KnowLine-26B-A4B-Gen2 (revision 284787c8b35c7d01da431be44ec3782db16a0704): PelaAI's 26B-A4B decision model (a LoRA-merged Gemma 4 26B A4B mixture of experts). Give it a state and typed questions (choice, yes/no, score) and it returns a probability for every option without generating text. Built and served by Midium's System One runtime (POST /v1/systemone).

What was quantized

• Language model: affine 8-bit, group size 64, round-to-nearest (no calibration data).
• Kept as in the source: the vision tower; the readout temperature folded into the final norm and final_logit_softcapping (24.0) is unchanged.
• Tokenizer, chat template and the decision configuration are unchanged.

Weights: 28.0 GB (bf16 source: 51.6 GB).

Evaluation against the bf16 model

Same runtime, same requests, question by question, on a seeded sample of up to 150 questions per set.

Suite	Questions	Top answer agrees with bf16	Accuracy change per set	Answers lost / gained
14 decision sets (routing, reply checks, tool preselection, model routing)	2,058	98.6–100%	-0.7 to +1.3 pts	3 / 4
Public sets, 150-way intent choice, 0–9 scores, option-shuffle and typo robustness	1,720	94.0–100%	-0.7 to +2.0 pts	5 / 8

Mean total-variation distance between the 8-bit and bf16 answer distributions: at most 0.027 per set.

Usage

This checkpoint keeps the source's decision protocol — KnowLine's label-token protocol (knowline_server.py): one prefill per question, a softmax over the option labels' logits (computed in fp32, as with SGLang's --enable-fp32-lm-head). It needs a runtime that implements that protocol: Midium's System One runtime loads it directly; generic MLX text-generation loaders do not.

License

Apache-2.0, as the source model.