vectionlabs/Salience-27B-R6-i1-GGUF

Salience 27B R6 — GGUF (imatrix)

llama.cpp quantizations of vectionlabs/Salience-27B-R6 built with an importance matrix and per-tensor bit allocation.

file	size	bits/param	KL vs BF16
Salience-27B-R6.i1-Q4_K_M.gguf	15.7 GB	4.87	—
Salience-27B-R6.i1-IQ4_XS.gguf	14.4 GB	4.44	—

imatrix.dat and calibration-corpus.txt are both in this repo. Every file here is reproducible from them with stock llama-quantize — nothing proprietary is involved and you do not have to take the numbers on faith.

What was actually done

The importance matrix. llama-imatrix measures which weights move the output on a calibration corpus; llama-quantize then spends its bit budget on those instead of spreading it evenly. The gain is largest at Q4 and near zero at Q6 and above.

The corpus is code, prose and tool-call transcripts. Not wikitext. On this architecture tool calling is the first category to break under quantization and has the widest error distribution — a handful of tokens carry most of the divergence. Calibrating on prose and hoping is how a quant ends up chatting fine and failing inside an agent loop.

Per-tensor allocation on the _XL files. attn_q/attn_k, ffn_down, and the token_embd/output/attn_gate group are quantized above the nominal tier; everything else stays at it. Those targets are not guesses — they come from a published per-tensor KL sweep on this architecture, which also found two things worth stating because they are counterintuitive: protecting attn_v has zero measurable effect, and protecting ssm_alpha makes the combined model worse. Neither is protected here.

The MTP head is pinned. Ordinary calibration text never exercises it, so it carries little imatrix coverage and the aggressive tiers refuse to quantize it without an explicit pin. It gets one.

What was NOT done

No quantization-aware training, no fine-tuning on the calibration set, and no claim of parity with any vendor's proprietary dynamic scheme. This is stock llama.cpp tooling with a corpus and an allocation table, both published above.

Run

llama-server -m Salience-27B-R6-Q4_K_M.gguf \
  --jinja --reasoning-format deepseek \
  -c 32768 -ngl 999

--jinja is not optional for agent use: it applies the model's own chat template, which turns the XML tool calls into proper OpenAI-style tool_calls and makes the model's reasoning_effort default take effect. Without it you get malformed calls and stock behaviour.

mmproj-*.gguf is the vision encoder — required for image input, ignored for text-only use. Pass it with --mmproj.

Which file

you have	take	note
48 GB+	Q8_0, or Q6_K	effectively lossless
32 GB	Q5_K_M
24 GB	Q4_K_M	the default — start here
16 GB	IQ4_XS, else Q3_K_M	tight once you add context
12 GB	IQ3_M	quality is visibly down
10 GB	IQ2_M, or Q2_K	read the next section first
8 GB	IQ2_XXS	read the next section first
CPU / unified memory	Q4_K_M	the whole model is read once per token, so file size is speed here

This is a dense model. There is no MoE router to corrupt, so the Q5/Q6 floor that applies to sparse models does not apply — Q4_K_M is a legitimate choice, not a compromise.

KV cache is cheaper than the parameter count suggests: hybrid attention means the cache only grows on 16 of 64 layers. At 32K with --cache-type-k q8_0 --cache-type-v q8_0 that is about 1 GB.

About the bottom of the ladder

Q2_K and the IQ2_* files are here because people ask for them and because the alternative is not running the model at all. They are not recommended, and two things are worth knowing before you pick one.

A 27B at Q2_K is not obviously better than a 9B at Q5_K_M, which is a similar file size. If a 9B can do your task, measure it before assuming the larger parameter count wins at this bit depth.

Expect the gap to widen with context length rather than stay flat. 48 of this model's 64 layers use linear attention with a recurrent state. Error introduced there accumulates along the sequence instead of staying local — a different failure shape from a plain transformer, where quantization error stays roughly per-token. The KL figures above are measured on short sequences and will understate this. If a low tier degrades faster at long context than the number suggests, that is the mechanism.

_{Built on Qwen3.8 (Apache-2.0).}