hoatac/gemma-4-E4B-it-AWQ

Gemma 4 E4B IT — AWQ 4-bit (Text-Only)

This repository contains an llmcompressor AWQ-style 4-bit quantization of google/gemma-4-E4B-it. All Linear layers (including vision and audio towers) are quantized, so this checkpoint is best suited for text-only workloads where the multimodal towers are never invoked.

For a multimodal variant where the vision and audio towers are preserved in FP16, see hoatac/gemma-4-E4B-it-AWQ-mm.

Quantization Summary

Field	Value
Method	llmcompressor (compressed-tensors format, WNA16 kernel)
Bit-width	4-bit weights, 16-bit activations (W4A16)
Group size	128
Symmetry	Symmetric
Calibration	32 mixed Turkish + English sentences (16 Turkish, 16 English) (general knowledge, technical, conversational)
Targets	All Linear layers
Preserved in FP16	lm_head

Validation

Smoke-tested with vLLM 0.29.0 (vllm/vllm-openai:v0.29.0-cu129):

• Model loads cleanly with --dtype float16
• compressed-tensors quantization is auto-detected from config.json
• Generates coherent Turkish and English completions
• Runs in text-only mode on a single RTX 3060 12GB with ~0.8 GiB KV cache (≈7 concurrent 2K-token requests)

Usage

With vLLM (recommended for production)

docker run --gpus all \
  -p 8000:8000 \
  -v $(pwd)/model:/model \
  --ipc=host \
  vllm/vllm-openai:v0.29.0-cu129 \
  --model /model \
  --served-model-name gemma-4-awq \
  --dtype float16 \
  --max-model-len 2048 \
  --max-num-seqs 10 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --enforce-eager

Do not pass --quantization awq. The checkpoint is in compressed-tensors format; vLLM detects it from config.json automatically.

The VLLM_WSL2_ENABLE_PIN_MEMORY=1 env var is required only when running vLLM inside Docker Desktop on WSL2. It fixes the RuntimeError: UVA is not available issue on that platform.

With Transformers

from transformers import AutoModelForConditionalGeneration, AutoTokenizer

model = AutoModelForConditionalGeneration.from_pretrained(
    "hoatac/gemma-4-E4B-it-AWQ",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("hoatac/gemma-4-E4B-it-AWQ", trust_remote_code=True)

messages = [{"role": "user", "content": "Merhaba, nasılsın?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Size

• On-disk: ~8.7 GB (model.safetensors)
• VRAM at load: ~8.9 GB (weights) + KV cache

Caveats

• Text-only checkpoint. Vision and audio towers are quantized, so multimodal inputs (images, audio, video) are not supported. For multimodal capability, use hoatac/gemma-4-E4B-it-AWQ-mm.
• Concurrent capacity: Expect ~7 concurrent 2K-token requests on a 12GB GPU with --max-model-len 2048.

License

This checkpoint inherits the Gemma license from the original model. Please review and accept the upstream license on Hugging Face before use.

Citation

@misc{atac2026gemma4awq,
  author       = {Ataç, Hasan Onur},
  title        = {Gemma 4 E4B IT -- AWQ 4-bit (Text-Only)},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/hoatac/gemma-4-E4B-it-AWQ}},
  note         = {Hugging Face Model Repository},
  orcid        = {0000-0003-1493-2063}
}