dekthedev/Qwen2.5-VL-3B-Instruct-GGUF

Qwen2.5-VL-3B-Instruct — GGUF

Quantized by @dekthedev.

All quantized GGUF variants of Qwen/Qwen2.5-VL-3B-Instruct. Quantized with llama.cpp for local and edge deployment.

Files

Variant	File	Size	Quality	Notes
Q2_K	qwen25vl-3b-q2_k.gguf	1.27 GB	Low	Smallest, fastest inference
Q4_K_S	qwen25vl-3b-q4_k_s.gguf	1.83 GB	Good	4-bit, small variant
Q4_K_M	qwen25vl-3b-q4_k_m.gguf	1.93 GB	Very good	4-bit, medium variant — best balance
Q5_K_M	qwen25vl-3b-q5_k_m.gguf	2.22 GB	Excellent	5-bit, sharper outputs
Q8_0	qwen25vl-3b-q8_0.gguf	3.29 GB	Near-lossless	8-bit, closest to full precision
mmproj F16	qwen25vl-3b-mmproj-f16.gguf	1.34 GB	—	Vision projector, required for image input

Usage with llama.cpp

Text only:

llama-cli \
  --model qwen25vl-3b-q4_k_m.gguf \
  --prompt "Hello! Who are you?" \
  --n-predict 128

With image:

llama-cli \
  --model qwen25vl-3b-q4_k_m.gguf \
  --mmproj qwen25vl-3b-mmproj-f16.gguf \
  --image your_image.jpg \
  --prompt "<|im_start|>user\nDescribe this image.<|im_end|>\n<|im_start|>assistant\n" \
  --n-predict 256