surendirakrishna/salad

Modified Qwen 3.5 4B VL — vision-boost α=1.25 (task-arithmetic merge)

Untested. This is a hypothesis to evaluate, not a validated improvement. Treat the unmodified Qwen 3.5 4B VL checkpoint as the control in every comparison.

What this is

A task-arithmetic merge of a vision-tuned checkpoint over its Qwen 3.5 base:

W_out = W_qwen + α · (W_tuned − W_qwen)

    model.visual.*   α = 1.25     vision tower, amplified
    everything else  α = 1.00     identical to the vision-tuned source

The language model, MTP head and embeddings are bit-identical to the vision-tuned source. Only the 297 vision-tower tensors (≈333 M params) differ.

Why the vision tower

Two facts from a full tensor-by-tensor diff of the vision-tuned source against Qwen 3.5:

• The source adapted the vision tower least — 1.49 % relative drift, versus 9.08 % for the language model. Its vision LayerNorms moved 0.04 %, i.e. were effectively frozen during fine-tuning.
• The source's weakest benchmark by a wide margin is old_scans at 51.1, about 31 points below the next-worst task (multi_column 82.1; baseline 99.9).

Degraded-scan failure is an image-quality problem, which the vision tower owns. The worst-performing capability and the least-trained component line up. This merge pushes the vision tower 25 % further along the direction training already moved it, on the theory that it was under-trained rather than mistrained.

This is a correlation. Extrapolating a task vector can just as easily degrade the model. That is what the eval is for.

Provenance

Base	MergeKit/Qwen — Qwen3.5, Qwen3_5ForConditionalGeneration
Fine-tune	Vision-tuned source checkpoint
Tool	MergeKit/merge_task_vector.py
Recipe	merge_recipe.json in this directory — per-tensor α, dtype, shape

The merge tool round-trips bit-exactly at α=1 (reproduces the tuned source) and at α=0 (reproduces Qwen), verified on 357 tensors; fractional α verified bit-exact against recomputation on 429 tensors across shard boundaries.

Two deliberate differences from the stock source files:

• The 48 SSM tensors are stored as F32, not BF16. The source downcast linear_attn.A_log and linear_attn.norm.weight from Qwen's float32. Since config.json sets mamba_ssm_dtype: float32, this merge restores the base model's precision instead of propagating the downcast.
• lm_head.weight is a copy of the merged embeddings. tie_word_embeddings is true, and the source lm_head is bit-identical to its own embed_tokens, so the tie is preserved explicitly.

Requirements

Roughly 10.6 GB of weights in bf16. A 16 GB card is workable; 24 GB is comfortable. This will not run on the machine it was built on (4 GB).

Running it

The processor, tokenizer, chat template and config.json are copied from the vision-tuned source, so it loads like the original Qwen 3.5 4B VL checkpoint:

pip install transformers accelerate
python your_vl_runner.py input.pdf --model_path /path/to/this/dir

or with vLLM:

vllm serve /path/to/this/dir --served-model-name modified-qwen-3.5-4b-vl

What to measure

Run olmOCR-bench against both this model and the unmodified Qwen 3.5 4B VL checkpoint, using the same decode settings and seed:

Task	Reference baseline	What to watch
old_scans	51.1	the target — does it move?
overall	85.8	must not regress
baseline	99.9	canary; any drop means the merge broke something
arxiv_math	86.9	canary for language-side damage

The single question: does old_scans improve without overall or baseline regressing? A gain on old_scans paid for by a drop elsewhere is not a win.

Check baseline first. If it falls below ~99, the extrapolation has damaged the model and there is no point reading the rest.

If it helps

Sweep α to find the peak — regenerate on the build machine:

python merge_task_vector.py --alpha "vision=1.15" --out ./vb-1.15
python merge_task_vector.py --alpha "vision=1.50" --out ./vb-1.50

Past roughly α=1.5 task-vector extrapolation usually degrades models sharply, so treat that as the ceiling. If the curve is still rising at 1.5, the honest conclusion is that the vision tower needs actual training on degraded scans, not more extrapolation.

If it does not help

The cheaper and more reliable fix for old scans is preprocessing — deskew, denoise, CLAHE contrast, adaptive (Sauvola) binarisation before the image reaches the model. On degraded scans that typically recovers more than any weight edit, and it costs no training and no GPU.