Instructions to use ThakiCloud/Qwen3-Coder-30B-A3B-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ThakiCloud/Qwen3-Coder-30B-A3B-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ThakiCloud/Qwen3-Coder-30B-A3B-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ThakiCloud/Qwen3-Coder-30B-A3B-W4A16")
model = AutoModelForCausalLM.from_pretrained("ThakiCloud/Qwen3-Coder-30B-A3B-W4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ThakiCloud/Qwen3-Coder-30B-A3B-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ThakiCloud/Qwen3-Coder-30B-A3B-W4A16

SGLang

How to use ThakiCloud/Qwen3-Coder-30B-A3B-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ThakiCloud/Qwen3-Coder-30B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ThakiCloud/Qwen3-Coder-30B-A3B-W4A16 with Docker Model Runner:
```
docker model run hf.co/ThakiCloud/Qwen3-Coder-30B-A3B-W4A16
```

Qwen3-Coder-30B-A3B-W4A16

W4A16 (4-bit weight) quantization of Qwen3-Coder-30B-A3B-Instruct. 61.1 GB → 16.69 GB (3.65×), with no detectable loss on HumanEval.

This is the control arm of a two-model release. Its sibling, Qwen3-Coder-30B-A3B-Prune3-W4A16, adds 3.1% expert pruning on top of the identical quantization recipe. We published both because the pair is the result: the 4-bit format is free here, and the pruning is what has a cliff.

Measured results

Single B200, vLLM, lm-eval humaneval_instruct via local-chat-completions, 150 questions, --gen_kwargs continue_final_message=True,add_generation_prompt=False.

model	size	HumanEval pass@1	stderr
Qwen3-Coder-30B-A3B-Instruct (bf16)	61.1 GB	0.9267	0.0214
this model (W4A16)	16.69 GB	0.9400	0.0195
FP8	29 GB	0.9400	—

At 150 questions the stderr is roughly 2 points, so 0.9400 against 0.9267 is not an improvement — read it as no detectable loss. That is the claim, and it is the only claim this eval supports.

What this costs you: throughput

The compression is a memory play, not a speed play. Measured on the pruned sibling against an FP8 baseline (single B200, vLLM, random dataset, input 512 / output 256, --ignore-eos):

concurrency	W4A16 (16.2 GB)	FP8 (29 GB)	FP8 lead
1	206.97 tok/s	284.66 tok/s	1.37×
10	1346.8 tok/s	1704.2 tok/s	1.27×
50	4169.99 tok/s	5974.8 tok/s	1.43×

Size note (added 2026-08-14). The FP8 baseline is recorded here as 29 GB from the 2026-08-06 run. The FP8 checkpoint currently in our registry measures 31.20 GB on disk. We have not confirmed the two are the same build, so both figures are left standing rather than one being quietly overwritten. Treat the FP8 size as approximate; the throughput ratios were measured against whichever build ran that day.

FP8 also holds better tail latency at concurrency 50 (322 ms vs 475 ms p99). 4-bit weights are dequantized at compute time while FP8 has native Blackwell tensor-core support, so smaller weights do not become faster compute here.

Choose this model to fit the model at all — onto fewer or smaller GPUs, or alongside a larger KV cache. If the model already fits your hardware and you want speed, use FP8, unless your hardware is Blackwell, in which case read on.

Speed versus NVFP4: this question is now answered

Earlier versions of this card said we had not tuned the 4-bit serving path and that an optimized kernel might narrow the gap to FP8. That hedge is now resolved, and not in W4A16's favor. A four-way comparison of bf16, FP8, W4A16, and NVFP4 — all on the same base model, same single B200, same workload (input 1746 tokens, output 256, vLLM 0.27.1, measured 2026-08-13) — found that the gap to FP8 does not close by tuning the W4A16 serving path. Kernel selection was not the missing piece.

It closes by switching quantization formats. NVFP4 quantizes activations as well as weights and runs directly on the Blackwell FP4 tensor cores, while W4A16 stores 4-bit weights and must dequantize them at compute time before every matmul. That is a structural difference between the two formats, not a tuning gap in this checkpoint.

concurrency	fp8	w4a16 (this model's format)	nvfp4	nvfp4 / fp8
1	257.6 tok/s	215.9 tok/s	303.3 tok/s	1.18x
32	3978.8 tok/s	3013.3 tok/s	5099.2 tok/s	1.28x
128	8130.3 tok/s	5427.7 tok/s	9941.8 tok/s	1.22x
512	6302.0 tok/s	4103.3 tok/s	6925.4 tok/s	1.10x

NVFP4 beats FP8 at every concurrency measured in the full sweep (1.07x to 1.32x) and beats W4A16 by 1.40x to 1.83x. The full sweep, tokens-per-joule numbers, and a HumanEval comparison across all four formats are on the NVFP4 sibling card: Qwen3-Coder-30B-A3B-NVFP4.

That does not make this checkpoint obsolete. W4A16 still wins on two things NVFP4 does not touch: it is smaller (16.70 GB against NVFP4's 18.11 GB), and it runs on Hopper (H200, H100), where NVFP4 has no native tensor-core path and falls back to Marlin emulation. If you need the smallest possible footprint, or your target is Hopper and NVFP4 is therefore not an option, this is the format to use. If your target is Blackwell and you want speed as well as a smaller footprint than bf16 or FP8, use the NVFP4 sibling instead.

On Hopper it is still a memory play (added 2026-08-15)

An earlier version of this card said "if your target is Hopper, this is still the right format," which read as a speed recommendation. It is not one, and we have now measured the Hopper case directly rather than inferring it. On a single H200 NVL, same workload as above (input 1746 / output 256, vLLM 0.27.1), this format loses to FP8 at every concurrency:

concurrency	FP8	W4A16 (this model)	W4A16 / FP8
32	2986.3 tok/s	2521.9 tok/s	0.84x
64	4441.2 tok/s	3616.6 tok/s	0.81x
128	5596.7 tok/s	4648.3 tok/s	0.83x

So the picture is consistent across both GPU generations: 4-bit weight-only storage buys memory, never throughput. On Blackwell the penalty is worse and grows with load (0.76x falling to 0.67x across the same band), because the faster the low-precision tensor cores are, the further a dequantize-before-you- compute path falls behind. Choose this checkpoint to make a model fit, not to make it fast.

Quality on Hopper matches Blackwell, as expected for the same weights: HumanEval pass@1 0.9390 and 0.9451 on two separate runs (n=164), against FP8's 0.9146 and 0.9207 on the same box. Rerunning an identical configuration moves the score by exactly one problem out of 164, so none of these four numbers is distinguishable from the others.

Both runs, the throughput ladder, and the per-hardware matrix this is drawn from are in our whitepaper ledger as 2026-08-15-per-hardware-quant-matrix.json.

Caveats

One model family, one code benchmark, 150 questions per task.
The throughput sweep above was run on the pruned sibling. The two are throughput-identical in our measurements, but this specific checkpoint was not separately swept.
Quantization recipe and calibration are in recipe.yaml in this repo.

Provenance

Quantized by ThakiCloud with llm-compressor. Base model Qwen/Qwen3-Coder-30B-A3B-Instruct is Apache-2.0 and not gated; this derivative inherits that license.