gemma-4-31b-it-int4-ov

Gemma4-31B-it INT4 OpenVINO — OVMS 2026.3 Deployment (Legacy VLM Pipeline)

This repository contains google/gemma-4-31B-it (dense 31B) exported to OpenVINO IR format with INT4 weight compression, ready to serve via OpenVINO Model Server (OVMS) 2026.3 as an OpenAI-compatible endpoint.

Validated on: Intel Arc Pro B60 (24 GB), Ubuntu 24.04, OVMS 2026.3.0 native CLI binary.

Read this before you deploy: this model must be served with OVMS's legacy VLM pipeline, not VLM_CB (continuous batching / paged attention). VLM_CB produces corrupted output for this model — see Known Challenges for how this was isolated. This is not a quantization bug; the INT4 export itself is clean.

---

Model Files

File	Size	Description
openvino_language_model.bin	~16 GB	INT4 (asymmetric) language model weights
openvino_vision_embeddings_model.bin	~551 MB	SigLIP2 vision encoder (unused for text-only inference, present because Gemma 4 is architecturally a VLM)
openvino_text_embeddings_model.bin	~1.4 GB	Text embedding model
openvino_tokenizer.xml/.bin	~17 MB	OV tokenizer
openvino_detokenizer.xml/.bin	~4.3 MB	OV detokenizer
graph.pbtxt	~1 KB	OVMS MediaPipe pipeline config (legacy VLM, not VLM_CB)

Quantization parameters: INT4_ASYM, group_size=128, ratio=1.0 (no mixed precision).

---

Quick Start

Prerequisites

• OVMS 2026.3 native CLI binary (tarball, not a Docker image — see below)
• Intel GPU with Level Zero / OpenCL drivers installed
• At least 18 GB GPU VRAM (Arc Pro B60 or equivalent)
• /dev/dri render device accessible
• Python 3.12 with the pyovms extension on PYTHONPATH (bundled with the OVMS tarball)

Step 1 — Get OVMS 2026.3

mkdir -p ~/ovms_2026.3
wget https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/2026.3.0.39717a69b/ovms_ubuntu24_2026.3.0_python_on.tar.gz
tar -xzf ovms_ubuntu24_2026.3.0_python_on.tar.gz -C ~/ovms_2026.3
~/ovms_2026.3/ovms/bin/ovms --version

This is the standalone binary release, not a docker save archive — don't docker load it.

Step 2 — Download the model

mkdir -p ~/ovms_models
cd ~/ovms_models
pip install -U "huggingface_hub[hf_xet]" hf_xet
export HF_XET_HIGH_PERFORMANCE=1
export HF_XET_NUM_CONCURRENT_RANGE_GETS=32

hf download shailesh83/gemma-4-31b-it-int4-ov \
    --local-dir google/gemma-4-31B-it \
    --exclude "*.md" ".gitattributes"

Step 3 — Create config.json

cat > config.json << 'EOF'
{
    "model_config_list": [
        {
            "config": {
                "name": "google/gemma-4-31B-it",
                "base_path": "google/gemma-4-31B-it"
            }
        }
    ]
}
EOF

Step 4 — Start OVMS (native CLI, GPU)

export PYTHONPATH=~/ovms_2026.3/ovms/lib/python:$PYTHONPATH

~/ovms_2026.3/ovms/bin/ovms \
  --config_path ~/ovms_models/config.json \
  --rest_port 9001 \
  --allowed_local_media_path ~/ovms_models \
  --allowed_media_domains all \
  --log_level INFO

The PYTHONPATH export is required — without it, OVMS's Python-Mediapipe backend can't find its bundled pyovms extension module and the server will not come up.

CPU-only fallback: change device: "GPU" to device: "CPU" in graph.pbtxt, then drop --device/--group-add if you're also using Docker.

Step 5 — Wait for the model to load

curl -s http://localhost:9001/v3/models

Wait for google/gemma-4-31B-it to show up in the response.

---

Testing

Text-only inference

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain how a transformer attention mechanism works, step by step."}],
    "max_tokens": 400,
    "temperature": 0.1
  }'

Streaming

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain photosynthesis briefly."}],
    "max_tokens": 200,
    "stream": true
  }'

Automated benchmark + corruption check

A benchmark script is included in this repo (benchmark_gemma4.py) that runs a fixed set of prompts, reports tokens/sec, and flags known corruption signatures (useful as a regression check after any OVMS/optimum-intel upgrade):

python benchmark_gemma4.py \
  --url http://localhost:9001/v3/chat/completions \
  --model google/gemma-4-31B-it \
  --runs 3

---

Benchmark Results

Single Intel Arc Pro B60, legacy VLM pipeline, single-stream, temperature=0.1. 4 prompt types x 3 runs = 12 requests.

Test	Max Tokens	Avg Elapsed (s)	Avg Tok/s	Result
attention_mechanism	400	20.41	19.60	3/3 clean
backprop_long	800	41.20	19.42	3/3 clean
math_check	100	5.25	19.04	3/3 clean
summary	300	15.27	19.64	3/3 clean

12/12 clean, 0 corrupted, 0 failed. Average 19.42 tok/s (range 18.84–19.66).

---

Known Challenges

This deployment went through a fairly involved debugging process. Summarized here so you don't have to repeat it:

1. INT4 corruption (red herring). Early attempts at exporting this model with INT4 compression (various group-size/ratio/symmetric combinations, across multiple optimum-intel branches) produced output that looked corrupted — garbled sub-tokens collapsing into repeated-token loops after roughly 100 tokens of otherwise coherent generation. This pattern held across CPU and GPU, across export methods, and across several quantization recipes, strongly suggesting an INT4-specific bug.
2. The actual root cause: OVMS's continuous-batching pipeline, not quantization. The same INT4 export produces clean output when tested directly via optimum-intel in Python and clean output when served through OVMS's legacy VLM pipeline — but corrupts when served through VLM_CB (continuous batching / paged attention), even with identical weights. This isolates the bug to VLM_CB's interaction with Gemma 4's hybrid attention head dimensions (256 sliding-window / 512 global) — likely a similar issue to one that required mainline vLLM to force a specific attention backend for this same model.
3. Gemma 4 is registered strictly as a VLM task, even for text-only use — exports must use --task image-text-to-text (direct optimum-cli) or --pipeline_type VLM/VLM_CB (export_model.py); text-generation-with-past is rejected outright.
4. Dependency conflicts are the biggest time sink. optimum-intel's Gemma 4 support currently lives on a fork (rkazants/optimum-intel@support_gemma_4), not mainline, and it has a narrow compatible window with transformers/safetensors/optimum-onnx versions. Installing OVMS's own export_model.py requirements.txt will silently install a different optimum-intel on top of the fork. Install requirements.txt first (full dependency resolution), then the fork second, unconstrained — this way the fork wins the conflict and pulls in compatible dependency versions.
5. OVMS's native GGUF loader does not support this model. If you already have a GGUF quant of Gemma 4 (e.g. from Unsloth), OVMS's built-in --pull --gguf_filename path will reject it — its recognized-architecture list includes gemma3 but not gemma4 as of this OVMS version. Use the IR export path in this repo instead.

---

Export Command (for reference)

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/common/export_models
pip install -r requirements.txt

pip install "git+https://github.com/rkazants/optimum-intel.git@support_gemma_4" \
    --extra-index-url https://download.pytorch.org/whl/cpu

python export_model.py text_generation \
    --source_model google/gemma-4-31B-it \
    --pipeline_type VLM \
    --weight-format int4 \
    --extra_quantization_params "--group-size 128 --ratio 1.0" \
    --target_device GPU \
    --model_repository_path ./ovms_models \
    --config_file_path ./ovms_models/config.json

Do not substitute --pipeline_type VLM_CB here — see Known Challenges.

---

Troubleshooting

Symptom	Cause	Fix
ModuleNotFoundError: No module named 'pyovms'	OVMS's Python-Mediapipe backend can't find its bundled extension module.	export PYTHONPATH=<ovms_dir>/ovms/lib/python:$PYTHONPATH before starting ovms.
KeyError: 'gemma4' in transformers	Active transformers version doesn't recognize the gemma4 model_type.	Install a transformers version with gemma4 support (5.13.0 confirmed working).
ValueError: ... only supports the tasks image-text-to-text for gemma4	This optimum-intel branch registers Gemma 4 strictly as a VLM task.	Use --task image-text-to-text or --pipeline_type VLM/VLM_CB, never text-generation-with-past.
ModuleNotFoundError: No module named 'optimum.exporters.onnx'	The optimum-onnx package got uninstalled or never installed.	pip install --no-deps optimum-onnx
ImportError: safetensors>=0.8.0 is required ... found safetensors==0.7.0	Version constraint wasn't quoted in shell, or --no-deps skipped the bump.	pip install "safetensors>=0.8.0" — quote it, or > gets parsed as a shell redirect.
Clean output via raw Python, corrupted via OVMS serving	VLM_CB pipeline bug with this model's hybrid attention heads.	Export and serve with --pipeline_type VLM (legacy) instead.
Invalid request URL	Using /v1/ instead of /v3/.	Change URL to /v3/chat/completions.

---

Hardware Requirements

Component	Minimum	Tested
GPU VRAM	18 GB	Intel Arc Pro B60 (24 GB)
System RAM	32 GB	256 GB
Storage	25 GB free	NVMe SSD
OS	Ubuntu 22.04+	Ubuntu 24.04
Driver	Intel GPU driver with Level Zero	Latest production

---

Known Limitations

• No continuous batching. The legacy VLM pipeline serves one request at a time; this configuration is validated for single-request serving only. Concurrent throughput is unvalidated.
• VLM_CB should not be used with this model until the underlying paged-attention issue with Gemma 4's hybrid head dimensions is fixed upstream in OpenVINO GenAI.
• Vision tower is present (SigLIP2) but unexercised by the text-only testing in this repo — image+text inference is architecturally supported but not validated here.

License

Gemma model weights are subject to the Gemma Terms of Use. This repository contains only the OpenVINO-converted weights with no modifications to model parameters.