qwen2vl-cord-lora
A QLoRA adapter fine-tuning Qwen2-VL-7B-Instruct for structured JSON extraction from receipt photos, trained on CORD-v2.
Part of the edge-vlm-extraction project β a methodology for fine-tuning and honestly measuring a vision-language model under edge hardware constraints.
Results β before vs after (same 40 held-out test examples)
| Metric | Base model | Fine-tuned | Ξ |
|---|---|---|---|
| Valid JSON rate | 7.5% | 100% | +92.5 pts |
| Field exact-match | 0% | 6.2% | +6.2 pts |
| Menu-item recall | 0% | 7.5% | +7.5 pts |
Reading the numbers honestly. The jump to 100% valid JSON is the core result. Field exact-match and recall remain low because 3 epochs on 600 examples teaches structure, not value memorisation. Both numbers are reported deliberately.
Training dynamics. Loss: 1.69 β 0.16 over 220 steps. Under 45 min on RTX 4090.
Training setup
- Base model: Qwen/Qwen2-VL-7B-Instruct
- Method: QLoRA β 4-bit NF4, LoRA on attention + MLP projections
- Trainable parameters: ~40M / 8.33B (0.48%)
- Dataset: CORD-v2, 600 train examples, 3 epochs
- Hardware: 1x RTX 4090 (24 GB VRAM)
Usage
from peft import PeftModel
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
base = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-7B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "saliousk/qwen2vl-cord-lora")
processor = AutoProcessor.from_pretrained("saliousk/qwen2vl-cord-lora")
Links
- Code: https://gitlab.com/saliousowce4/edge-vlm-extraction
- Author: Abdoulaye Saliou Seck
- Downloads last month
- 41
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support