lavendercomputing/Holo4-27B-GSQ-RCO-GGUF

Holo4-27B — GSQ-RCO-style GGUF

Per-tensor mixed-precision GGUF builds of the stock Hcompany/Holo4-27B weights, made for llama.cpp.

What this model is

Hcompany/Holo4-27B (developer: H Company) is a 27B vision-language model for Computer Use ("Qwen3.8 dense", built on Qwen/Qwen3.8-27B). "Dense" means no routed experts, but the network is a hybrid: 64 blocks — 16 full-attention + 48 linear-attention/GDN (attn_qkv, attn_gate, ssm_out, ssm_conv1d, ssm_alpha/beta), dense MLPs everywhere (17,408 hidden), 248,320-token vocab, 262,144-token context. Vision input uses a separate BF16 mmproj projector.

License: CC BY-NC 4.0 (non-commercial; the base Qwen3.8-27B is Apache-2.0 and both licenses ship with the vendor repo).

Files

File	Size	BPW	Ladder (64-block edge-weighted)
quant/Holo4-27B-GSQ-RCO-IQ2_XXS.gguf	7.27 GiB	2.32	64 blocks @ IQ2_XXS; embeddings Q4_K
quant/Holo4-27B-GSQ-RCO-IQ2_S.gguf	7.82 GiB	2.50	25 edge @ IQ2_S, 39 inner @ IQ2_XXS; embeddings Q4_K
quant/Holo4-27B-GSQ-RCO-IQ3_XXS.gguf	9.35 GiB	2.99	10 edge @ IQ3_XXS, 39 mid @ IQ2_S, 15 inner @ IQ2_XS
quant/Holo4-27B-GSQ-RCO-IQ3_S.gguf	10.96 GiB	3.50	19 edge @ IQ3_S, 16 mid @ IQ3_XXS, 29 inner @ IQ2_S
quant/mtp-Holo4-27B-GSQ-RCO-Q8_0.gguf	441 MiB	—	MTP draft head (Qwen3.8-27B base) for speculative decoding

Only the final quantized builds and the MTP draft head are hosted here. The source checkpoint and the importance matrix are linked below instead of being mirrored in this repository.

Source and provenance

Hcompany BF16 master
   → BF16 GGUF conversion (no intermediate quantization)
   → these 2.3 / 2.5 / 3.0 / 3.5 bpw builds

• Source: Hcompany/Holo4-27B — vendor BF16 safetensors, converted directly to BF16 GGUF for these builds.
• Importance matrix: mradermacher/Holo4-27B-i1-GGUF → Holo4-27B.imatrix.gguf — built against the stock weights. Coverage: 496 of 498 significant 2-D matrices; token_embd and output have no rows and use K-quants instead.
• Vision projector: Hcompany/Holo4-27B-GGUF → mmproj.f16.gguf — BF16, required for image input.

Quantization

"GSQ-RCO-style" means a format-compatible equivalent of the GSQ/RCO 3-bit recipe: the published GSQ/RCO engines emit only HF compressed-tensors and target sm_90+ GPUs, so they cannot produce GGUF directly.

The allocation spans every 2-D matrix of a block: dense MLPs, attention, and the GDN ssm_out projection. Edge blocks (by distance from block 0/63) take the top rung; redundant inner blocks take the bottom rung. Fixed policies:

• 3.50 tier: tied embeddings at Q8_0.
• 3.00 tier: embeddings at Q6_K — the vocab is 9.4% of parameters, and Q8 embeddings alone would consume 26% of the budget.
• 2.50 / 2.30 tiers: embeddings at Q4_K; the budget left for the block ladder is too small for Q6_K at these sizes.
• 1-D norms/biases fp32.

All four builds were smoke-tested for load and coherent output with CPU-only llama.cpp tooling. The 2-bit files are aggressive by design; use IQ3 for quality-sensitive work.

Usage

llama-cli -m quant/Holo4-27B-GSQ-RCO-IQ3_S.gguf \
  -p "Explain mixed-precision quantization." -ngl 99

# vision
llama-mtmd-cli -m quant/Holo4-27B-GSQ-RCO-IQ3_S.gguf \
  --mmproj Holo4-27B.mmproj-f16.gguf \
  --image screenshot.png -p "Describe this screen."

Speculative decoding (MTP)

The vendor Holo checkpoint declares an MTP head but ships no mtp.* weights, so these files have no usable built-in MTP layer. The draft head here is the MTP block from the matching base model (Qwen/Qwen3.8-27B), quantized to Q8_0. Tokenizer and dimensions match the Holo weights exactly, and Q8_0 acceptance was identical to BF16 in testing.

llama-server -m quant/Holo4-27B-GSQ-RCO-IQ3_S.gguf \
  -md quant/mtp-Holo4-27B-GSQ-RCO-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 -ngl 99

Measured greedy CPU acceptance on a code prompt: 86.5% (mean accepted length 3.50). The draft file has no token_embd.weight of its own; load it with -md, not as a standalone model. A borrow_shared_tensor log line is expected and harmless.

Honest limits

• License: CC BY-NC 4.0 — not redistributable commercially.
• These builds come directly from the vendor BF16 master; no intermediate quantization step is involved.
• Vision support was not smoke-tested here; the text-side tests used plain completion prompts.
• The MTP draft is the base-Qwen head, not a Holo-trained head; it is a generic draft for this architecture, validated against these weights.