gemma-2-9b-awq

AWQ (W4A16_ASYM) 4-bit quantization of Google's Gemma-2-9B, calibrated with llmcompressor, served with vLLM.

5.7 GB on disk · ~6 GB VRAM at load · 75 RPS on a single RTX 4090 · 200+ concurrent users · $0.03 per million tokens

I built this because I wanted to know: Can a 9B parameter model actually run in production on consumer hardware? The answer, after a lot of trial and error with CUDA OOMs, embedding dtype bugs, and locust async pitfalls, is a very solid yes — but only if you get the quantization right and understand where your bottleneck actually is.

---

Table of Contents

• Quick Start
• Performance
• Cost Economics
• Understanding the Bottleneck
• Quantization Details
• Bugs I hit so you don't have to
• How to Use
• The Full Serving Stack
• Why AWQ Instead of Alternatives
• Known Limitations
• Who built this

---

Quick Start

pip install vllm torch transformers

from vllm import LLM, SamplingParams

model = LLM(
    model="zaid646/gemma-2-9b-awq",
    quantization="compressed-tensors",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(
    ["Explain why AWQ quantization outperforms GPTQ for production serving."],
    params,
)
print(outputs[0].outputs[0].text)

Or with Docker:

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:v0.24.0 \
  --model zaid646/gemma-2-9b-awq \
  --quantization compressed-tensors

---

Performance

All numbers measured on a single NVIDIA RTX 4090 (24 GB) on Vast.ai, serving 200-token outputs. Zero failures across every test run.

AWQ vs FP16 Baseline

Metric	FP16 Gemma-2 9B	AWQ Gemma-2 9B	Improvement
Model size (disk)	~18 GB	5.7 GB	3.2x smaller
VRAM (model only)	~18 GB	~6 GB	3x less
Throughput (3 users)	1.5 RPS	6 RPS	4x higher
TTFT avg (3 users)	94 ms	40 ms	2.4x faster
TTFT p95 (3 users)	720 ms	69 ms	10.4x faster
ITL avg	21 ms	10 ms	2.1x faster
Max concurrent users (stable)	~5 (KV-cache limited)	~200	40x
Max queued connections (no crash)	—	5,000+	—

Scaling Behavior

Users	RPS	TTFT avg	TTFT p95	ITL	Failures
50	23.8	40 ms	69 ms	10 ms	0
100	37.3	56 ms	89 ms	14 ms	0
200	47.8	470 ms	1.7 s	19 ms	0
300	74.2	1.5 s	2.3 s	27 ms	0
500	75.3	2.2 s	2.3 s	27 ms	0
800	75.3	3.0 s	5.2 s	27 ms	0
1,500	~75	queue builds	—	27 ms	0
3,000	~75	queue=2,836	—	27 ms	0
5,000	~75	queue=4,825	—	27 ms	0

Throughput caps at ~75 RPS (GPU compute bound for 200-token outputs at 27ms ITL). Beyond ~200 users, TTFT grows linearly with queue depth. The system never crashes — it degrades gracefully.

---

Cost Economics

Vast.ai RTX 4090: ~$0.45/hour

Metric	FP16 (RTX 4090)	AWQ (RTX 4090)	GPT-4o-mini (API)
Sustained throughput	1.5 RPS (5 users)	75 RPS (200+ users)	N/A
Tokens/second	~300	~15,000	N/A
$/M input tokens	$3.00	$0.03	$0.15
$/M output tokens	$3.00	$0.03	$0.60
Concurrent capacity	~5 users	200+ users	unlimited
Data leaves your machine	No	No	Yes

Bottom line: AWQ delivers 50x more tokens per dollar than FP16 on the same hardware, and 5x cheaper than OpenAI's cheapest model.

---

Understanding the Bottleneck

The most important finding of this project: We hit the compute ceiling (27ms ITL = GPU saturated) before the memory ceiling.

With AWQ using only 6 GB for weights, ~18 GB is free for KV cache. vLLM's PagedAttention manages this efficiently. The bottleneck is compute: each token takes 27ms to generate on the RTX 4090, and nothing short of better hardware (or FP8) can make that faster.

• Up to ~200 users: Throughput rises linearly with batch utilization
• 200-500 users: GPU saturated, queue builds, throughput plateaus at ~75 RPS
• 500-5,000 users: TTFT grows linearly with queue depth, ITL stays flat at 27ms

If your SLO requires TTFT < 500ms, this setup supports ~200 concurrent users. If TTFT < 2s is acceptable, ~500 users. Beyond that, you need more GPUs.

---

Quantization Details

• Algorithm: AWQ (W4A16_ASYM, group size 128)
• Calibration: 64 samples from ultrachat-200k via llmcompressor 0.12.1a20260701
• Output format: compressed-tensors (vLLM-native, no conversion)
• Base image: pytorch/pytorch:2.12.1-cuda12.6-cudnn9-runtime
• GPU cap during calibration: 11 GiB (CPU-offloaded ~7 GiB layers to avoid OOM)
• Calibration time: ~45 minutes on RTX 4090

The pipeline is sequential: process one transformer layer at a time, compute activation-aware scaling factors, quantize, offload. This keeps peak VRAM under 13 GB.

---

Bugs I hit so you don't have to

Bug 1: llmcompressor stable + Gemma-2 embedding dtype crash

RuntimeError: Expected tensor for argument #1 'indices' to have one of the
following scalar types: Long, Int; but got torch.cuda.FloatTensor

llmcompressor==0.12.0 passes float-typed input_ids to Gemma-2's embedding layer. Fix: Use a clean pytorch/pytorch base image (not vLLM's) with llmcompressor==0.12.1a20260701.

Bug 2: GraniteMoeParallelExperts import rename

The nightly renamed GraniteMoeParallelExperts to GraniteMoeExperts. Fix: One-line sed patch in Dockerfile.

Bug 3: CUDA OOM during AWQ calibration

device_map="auto" puts 18 GB model on 24 GB GPU, leaving ~6 GB for calibration buffers. Fix: max_memory={0: "11GiB", "cpu": "40GiB"} to reserve 13 GB for calibration.

Bug 4: async httpx silently fails in Locust

Locust's greenlet runtime doesn't provide an event loop for async httpx. Fix: Use synchronous requests with manual SSE stream parsing.

---

How to Use

With vLLM (recommended)

from vllm import LLM, SamplingParams

model = LLM(
    model="zaid646/gemma-2-9b-awq",
    quantization="compressed-tensors",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    max_num_seqs=128,
    enable_prefix_caching=True,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(["Your prompt here."], params)
print(outputs[0].outputs[0].text)

OpenAI-compatible API

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zaid646/gemma-2-9b-awq",
    "prompt": "Explain AWQ quantization.",
    "max_tokens": 512,
    "temperature": 0.7
  }'

Docker Compose (full stack)

git clone https://github.com/ZAID646/llm-serving-lab.git
cp .env.example .env
# Edit .env with your HF_TOKEN
docker compose up -d

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "zaid646/gemma-2-9b-awq",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("zaid646/gemma-2-9b-awq")

Note: compressed-tensors is optimized for vLLM. Transformers inference works but throughput is significantly lower.

---

The Full Serving Stack

This model is one component of the llm-serving-lab project:

Component	Description
Quantization container	Clean pytorch/pytorch base with llmcompressor nightly
vLLM	0.24.0 with compressed-tensors, FlashAttention-3, PagedAttention
Prometheus	Scrapes vLLM + DCGM GPU metrics on port 9090
Grafana	Auto-provisioned dashboard with TTFT, ITL, throughput, queue depth, GPU util panels
DCGM Exporter	GPU power, temperature, memory, utilization metrics
Locust	Distributed load testing with custom synchronous SSE streaming client
K3s HPA	Queue-depth autoscaling via vllm:num_requests_waiting

---

Why AWQ Instead of Alternatives

Method	Pros	Cons	Verdict
AWQ	Best 4-bit quality; vLLM native; Marlin kernels	Calibration needs GPU	Best for production
GPTQ	Widely supported; mature	Worse quality at 4-bit	Viable but measurably worse
GGUF	Edge/CPU deployment	Lower throughput; fragmented	Single-user/edge
FP8	Native GPU; faster compute	Quality varies by model	Worth evaluating
No quantization	Maximum quality	~5 users on 24 GB GPU	Not viable

---

Known Limitations

1. TTFT degrades past ~200 users — GPU compute is the bottleneck, not memory
2. vLLM-optimized format — Transformers inference is slower
3. Measurable precision loss — <1% MMLU degradation but not lossless
4. No speculative decoding — Not yet stable with compressed-tensors
5. Single GPU ceiling — 75 RPS is the limit of one RTX 4090
6. Power draw — 4090 pulls ~350W under sustained load

---

Who built this

Mohammed Zaid Hussain — ML System Engineer.

I make LLMs work in production.

• GitHub
• Project repo
• Portfolio

If you found this useful, star the repo. If you found a bug, open an issue — PRs welcome.