Qwen3.8-27B-AWQ-INT4

AWQ W4A16 quantization of Qwen/Qwen3.8-27B, produced with llm-compressor in compressed-tensors pack-quantized format.

Built and qualified on Intel Arc Pro GPUs under SGLang.

Size	18.2 GiB (from 55.6 GiB BF16)
Format	compressed-tensors / pack-quantized
Scheme	W4A16 asymmetric, group size 128
Vision tower	preserved, BF16
MTP head	preserved, BF16
Gated DeltaNet projections	quantized, see below
Architecture	Qwen3_5ForConditionalGeneration

What is quantized, and what is not

Qwen3.8-27B is a hybrid: 48 of its 64 decoder layers use Gated DeltaNet (linear_attn.*) and 16 use full attention.

Those DeltaNet layers hold three large projections per layer, in_proj_qkv at 100 MiB, in_proj_z at 60 MiB and out_proj at 60 MiB, which together are 10.36 GiB in BF16, 47 percent of everything read on a decode step. Some 4-bit builds leave them at full precision. This one quantizes them, which is what the model authors do in their own FP8 release: the official Qwen3.8-27B-FP8 checkpoint carries weight_scale_inv for exactly those three tensors and excludes only the small ones.

Left at BF16, matching the FP8 release's modules_to_not_convert: the vision tower, the MTP head, lm_head, embed_tokens, all norms, and the DeltaNet scalar gates in_proj_a and in_proj_b. Those gates are 96 wide before sharding, so a 32-element group would span a third of a row.

Result: Gated DeltaNet falls from 10.36 GiB to 2.73 GiB, and the whole artifact reads 14.19 GiB per decode step instead of 21.82.

How much of this is actually 4 bit

"INT4" covers a wide range of formats and this one sits at the conservative end.

stored at 4 bits	24.33 B parameters, 4.16 bits each once per group scales and zero points are counted
kept at BF16	3.45 B parameters, 12.4 percent of the model
overall	5.63 bits per parameter, 18.2 GiB

What stays at BF16: the embedding table, lm_head, every norm, the Gated DeltaNet scalar gates, the vision tower and the MTP head. Activations are BF16 throughout; only the weights are quantized, hence W4A16.

This is deliberately larger than a maximally compressed 4 bit build of the same model. The choices that make it larger, per group asymmetric scales at group 128, full precision embeddings and output head, and an untouched vision tower, are the ones that keep it close to the original in the checks above.

Footprint and throughput are therefore not comparable across quantization formats. A build that packs more of the model into fewer bits will be smaller and will usually decode faster on the same hardware, because decode at batch one is bound by how many bytes move per token. Which model is better for your use is answered by measuring quality on your workload, not by comparing file sizes or tokens per second between formats.

Quantization details

• Source: Qwen/Qwen3.8-27B BF16, not re-quantized from the FP8 release.
• Method: AWQ via llm-compressor, W4A16 asymmetric, group size 128.
• Calibration: 512 sequences of 1024 tokens, 60 percent code instructions from codeparrot/self-instruct-starcoder and 40 percent general instructions from HuggingFaceH4/ultrachat_200k, chat-template formatted.
• Pipeline: sequential, one decoder layer at a time (sequential_targets=["Qwen3_5DecoderLayer"]), so each layer is calibrated against the quantized output of the layers before it.

Group size 128 rather than 32 was a measured choice. Finer groups quadruple the scale and zero-point metadata that is read alongside the weights on every step, and the int4 matmul is slower on them. A group-32 build of this same model measured slower end to end despite holding fewer weight bytes.

Quality

Checked against the BF16 original on prompts spanning code, reasoning, factual recall and summarization. Where the two are comparable the quantized model agrees with BF16 on the most likely token at every step, and the BF16 token never falls outside the top eight candidates. Prefill perplexity moves from 8.87 to 9.33.

Greedy text still diverges from BF16, as it will for any 4 bit build. Treat this as a 4 bit model: good for interactive use, coding assistance and long context work, and not a drop in replacement where exact BF16 outputs matter.

Usage

SGLang on Intel Arc

docker run --rm -d --name qwen38 \
  --device=/dev/dri -v /dev/dri:/dev/dri \
  --group-add video --group-add "$(getent group render | cut -d: -f3)" \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --ipc=host --shm-size=64g --ulimit memlock=-1 \
  -p 30000:30000 -v /path/to/Qwen3.8-27B-AWQ-INT4:/model:ro \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  rahulunair/sglang-xpu:qwen3.8-27b-20260816 \
  python -m sglang.launch_server --model-path /model \
    --device xpu --tp-size 4 --host 0.0.0.0 --port 30000 \
    --trust-remote-code --attention-backend intel_xpu --page-size 64 \
    --context-length 32768 --max-total-tokens 32768 \
    --chunked-prefill-size 4096 --mem-fraction-static 0.85 \
    --cuda-graph-config '{"decode":{"backend":"full","bs":[1,2,4,8]},"prefill":{"backend":"disabled"}}' \
    --skip-server-warmup

SYS_PTRACE and seccomp=unconfined are required. Without them the symmetric-memory all-reduce declines silently and the slower stock collective runs, with nothing in the log to say so.

Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "Qwen3.8-27B-AWQ-INT4", device_map="auto", trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "Qwen3.8-27B-AWQ-INT4", trust_remote_code=True,
)

Limitations

• Calibration is text only. The vision tower is not quantized so it is unaffected, but no image data was used.
• Multimodal is preserved but lightly exercised. The vision tower and processor configs are present and the model loads as a conditional generation model; image inputs beyond basic checks have not been qualified.
• The MTP head is preserved but speculative decoding is not qualified.

License

Apache 2.0, inherited from the base model.