d3-flash — MLX 8-bit
An 8-bit MLX quantization of vllm-sr/d3-flash (revision 13dc1cc8ef23caefa765907e1d764751e5629f7d): vLLM Semantic Router's 9B multimodal decision model (Qwen3.5-9B + vision).
Give it a state and typed questions (choice, yes/no, score) and it returns a probability for every option without
generating text. Built and served by Midium's System One runtime (POST /v1/systemone).
What was quantized
- Language model: affine 8-bit, group size 64, round-to-nearest (no calibration data).
- Kept as in the source: the vision tower and the 255-way answer-code readout (
readout.safetensors). - Tokenizer, chat template and the decision configuration are unchanged.
Weights: 9.4 GB (bf16 source: 16.8 GB).
Evaluation against the bf16 model
Same runtime, same requests, question by question, on a seeded sample of up to 150 questions per set.
| Suite | Questions | Top answer agrees with bf16 | Accuracy change per set | Answers lost / gained |
|---|---|---|---|---|
| 14 decision sets (routing, reply checks, tool preselection, model routing) | 2,058 | 98.7–100% | -0.7 to +1.3 pts | 3 / 3 |
| Public sets, 150-way intent choice, 0–9 scores, option-shuffle and typo robustness, synthetic images | 1,860 | 97.5–100% | -0.8 to +0.0 pts | 2 / 1 |
All 140 synthetic image questions (colour, shape, count, position, printed word, tallest bar, presence) give the same answer as bf16.
Mean total-variation distance between the 8-bit and bf16 answer distributions: at most 0.009 per set.
Usage
This checkpoint keeps the source's decision protocol — the d3 code-readout protocol: the 255-code readout at the last token, with the full-attention layers attending to the whole prompt both ways (noncausal_full_attention). It needs a runtime that implements that
protocol: Midium's System One runtime loads it directly; generic MLX text-generation loaders do not.
License
Apache-2.0, as the source model.
- Downloads last month
- 20
8-bit