leonardosalem/qwen-vl-driving-multitask

Qwen-VL QLoRA adapters — unified multi-task driving model

LoRA adapters from a master's thesis on parameter-efficient adaptation of vision-language models for autonomous driving.

• Task: Single adapter handling grounding, VQA and trajectory via prompt switching
• Dataset: Talk2Car + LingoQA + CoVLA-mini (temperature-balanced mix)
• Metric: AP50 / LingoJudge / ADE@3s
• Method: QLoRA — 4-bit NF4 frozen base, LoRA rank 16, alpha 32, dropout 0.05, lora_target: all, paged_adamw_8bit, cosine schedule

Results

Subfolder	Base model	Checkpoint	Grounding AP50	VQA	ADE@3s
qwen3-4b	Qwen/Qwen3-VL-4B-Instruct	checkpoint-3200	79.12%	52.8 (4-bit) / 58.4 (bf16)	0.7047
qwen2.5-3b	Qwen/Qwen2.5-VL-3B-Instruct	checkpoint-1600	45.44%	53.4	0.8385

Figures are from full-split evaluations (2447 grounding / 500 VQA / 500 trajectory samples). A 50-sample evaluation of the same Qwen2.5-VL-3B checkpoint gives markedly different values (60.0% / 54.0 / 0.7707); prefer the full-split numbers above.

Unified training trades single-task accuracy for one model covering all three tasks.

Contents

Each subfolder is a self-contained PEFT adapter (weights, config, tokenizer and processor files). Training-resume artefacts (optimizer, scheduler, RNG state) are excluded.

• qwen3-4b/ — from results/checkpoints/qwen3-4b-multitask/checkpoint-3200
• qwen2.5-3b/ — from results/checkpoints/qwen2.5-3b-multitask/checkpoint-1600

Usage

from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base = "Qwen/Qwen3-VL-4B-Instruct"
model = Qwen3VLForConditionalGeneration.from_pretrained(base, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "leonardosalem/qwen-vl-driving-multitask", subfolder="qwen3-4b")
processor = AutoProcessor.from_pretrained(base)

Match the subfolder to its base model in the table above.

Reproducing

Training and evaluation code, configs and metrics: https://github.com/Leonardo-salem/thesis-vm-final

REPRODUCE.md in that repository gives the exact command, checkpoint, inference precision and sample count behind every number.