vectionlabs/Salience-27B-R6-GGUF

Salience 27B R6 — GGUF (static)

llama.cpp quantizations of vectionlabs/Salience-27B-R6, built with stock llama-quantize defaults and no importance matrix.

file	size	bits/param	KL vs BF16
Salience-27B-R6-Q4_K_M.gguf	15.7 GB	4.84	—
Salience-27B-R6-Q8_0.gguf	27.1 GB	8.36	—
Salience-27B-R6-Q6_K.gguf	20.9 GB	6.46	—

Static or imatrix?

For Q5_K_M and above the difference is small enough to ignore. Below that, take the imatrix build — that is where the bit budget actually needs steering: vectionlabs/Salience-27B-R6-i1-GGUF. At Q3 and Q2 it stops being a preference: the static files at those tiers are here for comparison, and the imatrix repo has IQ3_M, IQ2_M and IQ2_XXS, which cannot be built without one at all.

These exist for people who would rather have a file calibrated on nothing than a file calibrated on someone else's idea of representative text. That is a defensible preference and this repo respects it.

Run

llama-server -m Salience-27B-R6-Q4_K_M.gguf \
  --jinja --reasoning-format deepseek \
  -c 32768 -ngl 999

--jinja is not optional for agent use: it applies the model's own chat template, which turns the XML tool calls into proper OpenAI-style tool_calls and makes the model's reasoning_effort default take effect. Without it you get malformed calls and stock behaviour.

mmproj-*.gguf is the vision encoder — required for image input, ignored for text-only use. Pass it with --mmproj.

Which file

you have	take	note
48 GB+	Q8_0, or Q6_K	effectively lossless
32 GB	Q5_K_M
24 GB	Q4_K_M	the default — start here
16 GB	IQ4_XS, else Q3_K_M	tight once you add context
12 GB	IQ3_M	quality is visibly down
10 GB	IQ2_M, or Q2_K	read the next section first
8 GB	IQ2_XXS	read the next section first
CPU / unified memory	Q4_K_M	the whole model is read once per token, so file size is speed here

This is a dense model. There is no MoE router to corrupt, so the Q5/Q6 floor that applies to sparse models does not apply — Q4_K_M is a legitimate choice, not a compromise.

KV cache is cheaper than the parameter count suggests: hybrid attention means the cache only grows on 16 of 64 layers. At 32K with --cache-type-k q8_0 --cache-type-v q8_0 that is about 1 GB.

About the bottom of the ladder

Q2_K and the IQ2_* files are here because people ask for them and because the alternative is not running the model at all. They are not recommended, and two things are worth knowing before you pick one.

A 27B at Q2_K is not obviously better than a 9B at Q5_K_M, which is a similar file size. If a 9B can do your task, measure it before assuming the larger parameter count wins at this bit depth.

Expect the gap to widen with context length rather than stay flat. 48 of this model's 64 layers use linear attention with a recurrent state. Error introduced there accumulates along the sequence instead of staying local — a different failure shape from a plain transformer, where quantization error stays roughly per-token. The KL figures above are measured on short sequences and will understate this. If a low tier degrades faster at long context than the number suggests, that is the mechanism.

_{Built on Qwen3.8 (Apache-2.0).}