Mini-VLM (ONNX, for the browser)
LLaVA-style vision-language model: SigLIP-base image encoder + 2-layer MLP projector + Qwen2.5-0.5B-Instruct with LoRA adapters (merged), trained on COCO captions and 200K LLaVA conversations. These are the stage-2 (instruction-tuned) weights: 76.8% accuracy / 80.1 F1 on POPE. (The stage-1 projector-only checkpoint scores 109.0 CIDEr on COCO captions; stage 2 writes long descriptions, so caption metrics do not apply to it.)
Files used by the in-browser demo (Space):
| File | What it is |
|---|---|
vision.onnx |
SigLIP + projector, 8-bit weights (MatMulNBits). Input RGB in [0, 1], 1×3×224×224 → 196 image tokens (1×196×896) |
decoder.onnx |
Qwen2.5-0.5B with merged LoRA, 4-bit weights, built with onnxruntime-genai (exclude_embeds), takes inputs_embeds + KV cache, returns last-position logits |
embed_q8.bin, embed_scale.bin |
int8 token-embedding table with per-row float32 scales (looked up in JavaScript) |
meta.json |
prompt format, token ids, shapes and the export check |
Prompt: the image tokens replace 196 <|image_pad|> placeholders between <|vision_start|> and
<|vision_end|> in the first user turn of the Qwen chat template (see meta.json). Greedy
decoding with repetition penalty 1.1.