openvino-models

Pre-converted OpenVINO GenAI models, ready to use with ovtool.

Directory Layout

<category>/<model-name>/<format-quantization>/

• Category: llm (text generation) / vlm (vision-language multimodal) / image (diffusion image generation)
• Quantization naming: int4-asym-g128 = INT4 asymmetric, group 128; int4-sym-g128 = INT4 symmetric, group 128 (required for NPU); int8 = INT8 weight compression

Model Inventory

Every llm/vlm model ships the full five-variant ladder: int4-asym-g128 / int4-sym-g128 / int4-awq-g128 / int8 / fp16.

Model	Source (Hugging Face)	Devices
llm/Qwen2.5-0.5B-Instruct/*	Qwen/Qwen2.5-0.5B-Instruct	CPU / GPU (sym also NPU-ready)
llm/Qwen3-0.6B/*	Qwen/Qwen3-0.6B	CPU / GPU (sym verified on NPU ~21 tok/s; awq ~78 tok/s on iGPU)
llm/Qwen3-1.7B/*	Qwen/Qwen3-1.7B	CPU / GPU (sym verified on NPU ~16 tok/s; asym ~42 tok/s on iGPU)
llm/Qwen3-4B/*	Qwen/Qwen3-4B	CPU / GPU (sym verified on NPU ~9 tok/s; asym ~22 / awq ~24 tok/s on iGPU)
vlm/Qwen3.5-0.8B/*	Qwen/Qwen3.5-0.8B	CPU / GPU (text-only)
vlm/Qwen3.5-2B/*	Qwen/Qwen3.5-2B	CPU / GPU (text-only)
vlm/Qwen3-VL-2B-Instruct/*	Qwen/Qwen3-VL-2B-Instruct	CPU / GPU (text-only)
vlm/Qwen3-VL-4B-Instruct/*	Qwen/Qwen3-VL-4B-Instruct	CPU / GPU (text-only; ~24 tok/s int4 on iGPU)
image/SD-Turbo/{int8,int4-g64}	stabilityai/sd-turbo	GPU (t2i / i2i; g64 required: UNet channels 320 % 128 != 0)

Usage

pip install -e .   # install ovtool
ovtool chat  -m models/llm/Qwen3-0.6B/int4-sym-g128 -d NPU
ovtool image -m models/image/SD-Turbo/int8 -d GPU "a corgi surfing a wave" --steps 4

Known Limitations (2026-09, openvino-genai 2026.3.1)

• Image input for Qwen3-VL / Qwen3.5 hits a shape-compatibility issue in this GenAI release (fixed on master, pending release); text-only mode works fine
• NPU supports only symmetric INT4 models, and requires a fixed prompt/response budget at pipeline compile time (handled automatically by ovtool)
• Conversion environment: optimum-intel 2.1.0 + NNCF 3.3 + transformers 5.2.0