impacte/Qwen3.8-27B-GGUF

Qwen3.8-27B (Q4_K_M) — GGUF + llama.cpp config

Full-quality Qwen3.8-27B in GGUF format with a ready-to-run llama.cpp configuration that uses --no-kv-offload to keep the KV cache in system RAM — unlocking the full 256K context even on modest GPU setups.

---

DESCRIPTION

A dense 27.8B-parameter multimodal model from the Qwen team, packaged as a GGUF with a complete llama.cpp launcher. Unlike MoE models, every parameter is active on every token, giving maximum quality per token. Its hybrid attention architecture (48 linear + 16 full attention layers) delivers efficient long-context inference without the quadratic cost of pure attention.

Key features:

• Dense 27.8B — all parameters active, no MoE routing
• Hybrid attention — 48 linear + 16 full attention layers (3:1 ratio)
• 256K native context (262,144 tokens)
• Multimodal — text + image + video (via mmproj)
• Reasoning model — thinking blocks before response
• Tool calling — <tool_call><function=...> XML format
• Q4_K_M quantization — imatrix-calibrated, 17.77 GB

Architecture

Property	Value
Architecture	Qwen3.5 hybrid (linear + full attention, dense)
Total Parameters	27.78B (all active)
Layers	64 (48 linear + 16 full attention)
Hidden Size	5,120
Attention Heads	24 (4 KV heads)
Native Context	262,144 tokens (256K)
Modalities	Text + Image + Video
Quantization	Q4_K_M imatrix-calibrated
Model Size	~17.8 GB (weights)
License	Apache 2.0
Upstream	Qwen/Qwen3.8-27B

---

FILES

File	Purpose
Qwen3.8-27B-Q4_K_M.gguf	Quantized weights (17.77 GB)
mmproj-Qwen3.8-27B-f16.gguf	Multimodal projector (vision/video)
run-llamacpp.sh	llama.cpp server launcher with --no-kv-offload
Modelfile	Ollama Modelfile

---

REQUIREMENTS

Resource	Minimum	Recommended
GPU Memory	24 GB VRAM	32 GB+ VRAM
System RAM	64 GB	128 GB
Disk Space	20 GB free	50 GB+ free
llama.cpp	recent build (CUDA)	latest master
NVIDIA Driver	525+	550+

💡 Why --no-kv-offload? The 17.8 GB weights fit on GPU, but the KV cache for 256K context is huge. By keeping the KV cache in system RAM, all 64 layers stay on GPU while the context window can reach the full 256K — ideal for 24 GB VRAM + 128 GB RAM setups.

---

QUICK START (llama.cpp — recommended)

1. Download this repo

git lfs clone https://huggingface.co/impacte/Qwen3.8-27B-GGUF
cd Qwen3.8-27B-GGUF

2. Start the server

./run-llamacpp.sh
# -> OpenAI-compatible API at http://localhost:8082/v1

The script sets CUDA_VISIBLE_DEVICES=0,1 (device ordinals, not PCI IDs) and uses --no-kv-offload so the KV cache goes to system RAM.

3. Test it

curl http://localhost:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Hello!"}]}'

---

USAGE

Manual llama.cpp command

llama-server \
  --model Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-f16.gguf \
  -ngl 99 \
  --no-kv-offload \
  -c 262144 \
  --parallel 1

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8082/v1", api_key="none")

response = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Explain quantum computing simply"}],
)
print(response.choices[0].message.content)

Multimodal (image)

curl http://localhost:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<BASE64>"}}
      ]
    }]
  }'

---

RUN WITH OLLAMA

ollama create qwen3.8-27b -f Modelfile
ollama run qwen3.8-27b

Note: On 24 GB VRAM, Ollama cannot offload the KV cache to RAM, so the default context is limited to 8K. Use the llama.cpp script above for the full 256K context. A 32 GB+ VRAM variant is also published on Ollama as oamazonasgabriel/qwen3.8-27b:q4-km-32gbGPU.

---

MEMORY & PERFORMANCE

Component	Size
Model weights (Q4_K_M)	~17.8 GB
KV cache (in RAM, 256K ctx)	~20-28 GB (system RAM)
GPU VRAM used	~18-19 GB (all layers on GPU)

Performance (24 GB VRAM + 128 GB RAM):

• Prompt processing: ~60-120 tok/s (hybrid linear attention)
• Text generation: ~20-40 tok/s
• Model load time: ~60-90 seconds

---

TROUBLESHOOTING

Symptom	Fix
invalid device ordinal / no GPU	Set CUDA_VISIBLE_DEVICES=0,1 (device ordinals, not PCI IDs)
OOM on GPU	Ensure --no-kv-offload is set so KV cache goes to RAM
Slow generation	Check nvidia-smi — all layers should be on GPU
Vision not working	Ensure --mmproj mmproj-Qwen3.8-27B-f16.gguf is passed

---

CREDITS

Role	Entity
Base Model	Qwen Team, Alibaba Group
Original Model	Qwen3.8-27B
GGUF Quantization	bartowski — Q4_K_M imatrix-calibrated
llama.cpp Packaging	impacte.tech
License	Apache 2.0

---

LINKS

Resource	URL
This repo	https://huggingface.co/impacte/Qwen3.8-27B-GGUF
Upstream model	https://huggingface.co/Qwen/Qwen3.8-27B
bartowski quantized version	https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
Ollama model	https://ollama.com/oamazonasgabriel/qwen3.8-27b
llama.cpp	https://github.com/ggml-org/llama.cpp
Built by	impacte.tech