mxv456/qwen3-vl-2b-lora-map-routes

Qwen3-VL-2B + LoRA: walking routes on a map image, as a polyline

A LoRA adapter for Qwen/Qwen3-VL-2B-Instruct that answers "find the walking route from the green disc to the red disc" on a top-down map image with a JSON list of points in Qwen's 0–1000 grounding grid. It is the vision-language baseline of spatial-reasoning-from-scratch, trained by Marvin Uhlmann with his own training loop (qwen_fine_tuning/ in the repository).

Figure. Three De Pijp validation routes drawn by five methods: Qwen3-VL-4B zero-shot, this adapter (row 2), a 4-block transformer drawing the path in one pass, the iterative wave search, and the search with a cleanup pass. Buildings dark, start and goal as discs, the method's polyline or lit cells orange; verdicts by the polyline rule. The adapter's remaining errors are chords through buildings.

Results (polyline rule, 500 held-out routes)

model	solved	2–3 fronts	4–5	6+	largest failure
Qwen3-VL-2B zero-shot (200 routes)	2.5 %	6 %	2 %	0 %	wrong endpoints 92 %
Qwen3-VL-4B zero-shot (200 routes)	24.5 %	35 %	35 %	2 %	crosses a building 52 %
this adapter, 1k training routes	21.4 %	58 %	28 %	1 %	endpoints 39 %, crosses 36 %
this adapter, 5k training routes	31.0 %	74 %	38 %	8 %	crosses 47 %
this adapter, 19.3k training routes	49.6 %	81 %	62 %	28 %	crosses a building 36 %
straight line between the markers (floor)	27.6 %	73 %	36 %	2 %	crosses a building
iterative wave search, 9 M-parameter network, same rule	76.4 %	93 %	87 %	63 %	gap in the drawing

Fronts = the route's walking length in 40 m steps (2–3 ≈ 40–120 m, 6+ ≥ 200 m). The rule: start and end within 6 m of the markers, no segment crosses a building pixel, length at most 1.10 × the teacher's clearance-priced walking path.

Training

LoRA rank 16, α 32, dropout 0.05 on q_proj, k_proj, v_proj, o_proj of the language model (about 0.3 % of parameters), vision tower frozen; AdamW lr 2e-4, cosine with 5 % warmup, effective batch 4, 3 epochs, bf16, gradient checkpointing, on one RTX 3060 (about 8 h). Data: 19,309 routes of the De Pijp training store rendered at 2 px per metre (512 px; buildings dark, start green, goal red), target = the exact shortest walking path simplified to a polyline, in the 0–1000 grid; routes whose path leaves the window excluded. Prompt route_v1 and coordinate conversion in qwen_reference/ of the repository.

How to use

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
base = "Qwen/Qwen3-VL-2B-Instruct"
processor = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "mxv456/qwen3-vl-2b-lora-map-routes")
# prompt: qwen_reference/prompts.py route_v1 in the repository; the answer is a JSON list of [x, y] in a 0–1000 grid

The repository's qwen_fine_tuning/eval_lora.py runs the exact evaluation; the scorer is spatial_data/track_a_export.py and the route images are in the GitHub release's evaluation bundle.

Limitations

Trained and evaluated on one district of one city rendered in one style; not tested on illustrated or real-world map images; the dominant remaining error is a line through a building; no reinforcement learning.

Licence

The adapter is released under Apache-2.0, the licence of the base model. Map data licences: the repository's DATA_LICENSES.md.