tinyvlm-vqa

A minimal LLaVA-style VLM fine-tuned for visual question answering.

• Vision encoder: google/siglip2-base-patch16-224 (frozen)
• LLM: Qwen/Qwen2.5-0.5B-Instruct (LoRA fine-tuned)
• Projector: 2-layer MLP, trained in Stage 1, refined jointly in Stage 2
• Task: Visual question answering

Files

• projector.pt -- projector MLP weights (state_dict)
• lora_adapter/ -- PEFT LoRA adapter for the Qwen2.5 LLM

Usage

This repo does not contain a full merged model -- you need the companion model.py / inference.py code (from the training repo) to reload the architecture, then load these weights on top of the base checkpoints:

from model import TinyVLM
from peft import PeftModel
import torch
from huggingface_hub import hf_hub_download, snapshot_download

model = TinyVLM(freeze_vision=True, freeze_llm=True)

projector_path = hf_hub_download(repo_id="dhruvpatel93/tinyvlm-vqa", filename="projector.pt")
model.projector.load_state_dict(torch.load(projector_path, map_location="cpu"))

lora_path = snapshot_download(repo_id="dhruvpatel93/tinyvlm-vqa", allow_patterns=["lora_adapter/*"])
model.llm = PeftModel.from_pretrained(model.llm, lora_path + "/lora_adapter")

Training data

Stage 1: Flickr8k captions. Stage 2: a subset of VQAv2.

Limitations

Single image per example, 224x224 input resolution, small (~0.5B) base LLM -- expect casual/simple-VQA quality, not document-grade OCR/reasoning.