duxiaodan/teaching-an-agent-to-sketch-one-part-at-a-time

Teaching an Agent to Sketch One Part at a Time

Model release for the paper Teaching an Agent to Sketch One Part at a Time.

The agent draws a sketch one part at a time: at each step it is shown the caption and an image of the canvas so far, and it emits the next part's strokes as cubic-Bézier paths. This is the paper's final RL checkpoint (multi-turn GRPO with a DreamSim visual reward, RL step 1000, initialized from SFT step 5400), fine-tuned from Qwen/Qwen3-VL-30B-A3B-Instruct.

What's in this repo

Path	Contents
repo root	Full merged checkpoint (recommended): bf16 safetensors in exactly the base model's layout, directly loadable by transformers and servable by vLLM / SGLang with no LoRA flags.
adapter/	The raw LoRA adapter as exported from Tinker. Not loadable with PeftModel.from_pretrained as-is — see the adapter section before using it.

Using the merged model (recommended)

The merged checkpoint is a drop-in Qwen3-VL-30B-A3B model — anything that runs the base model runs this repo the same way.

transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "duxiaodan/teaching-an-agent-to-sketch-one-part-at-a-time"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(REPO)

vLLM

vllm serve duxiaodan/teaching-an-agent-to-sketch-one-part-at-a-time \
  --max-model-len 24576

SGLang

python -m sglang.launch_server \
  --model-path duxiaodan/teaching-an-agent-to-sketch-one-part-at-a-time \
  --mem-fraction-static 0.85

(Unrelated to this checkpoint: SGLang 0.5.9 on torch 2.9.1 needs SGLANG_DISABLE_CUDNN_CHECK=1 to start.)

Hardware sizing. The weights are ~58 GB of bf16 (30B-A3B MoE). All three stacks above were validated on a single 96 GB GPU (NVIDIA RTX PRO 6000 Blackwell). On GPUs of that size, cap the context (e.g. --max-model-len 24576 for vLLM) — the base model's default 256K context does not leave room for the KV cache. Sketch rollouts are short, so a ~24K context is more than enough.

All three stacks produced clean, near-identical sketches from this checkpoint in our validation (full 32-path rollouts with negligible off-canvas strokes).

Merged model vs. base + adapter

The merged checkpoint is functionally equivalent but not token-identical to applying the adapter on top of the base model in PEFT: merging LoRA deltas into bf16 weights introduces rounding at the merge, and the MoE router is sensitive to tiny logit perturbations, so long greedy generations can diverge after some tokens (first-step argmax is identical, and generated sketches are qualitatively the same). For reproducing the paper's behavior, the merged checkpoint at the repo root is the reference release.

The raw adapter (adapter/)

adapter/adapter_config.json + adapter/adapter_model.safetensors (LoRA r=64, lora_alpha=32, target_modules="all-linear" — attention, lm_head, and the MoE expert projections).

Honest caveats — read before using:

• Not PEFT-loadable as-is. The safetensors uses Tinker-internal tensor naming. The attention and lm_head entries can be renamed to standard PEFT keys, but the MoE expert deltas are stored as fused 3D tensors (all experts stacked), which PEFT cannot wrap at all — they can only be merged into the base weights. That merge is exactly what the checkpoint at the repo root is.
• vLLM / SGLang --enable-lora will not work correctly. Current LoRA serving paths (tested with vLLM 0.24 and SGLang 0.5.9) can apply only the attention subset of this adapter — roughly 0.4% of the fine-tune's total weight delta — and the resulting outputs are near-base-model garbage. Serve the merged model instead (no LoRA flags needed).

The adapter is published for provenance and for users who want to do their own merging or analysis of the weight deltas.

Training provenance

• Base model: Qwen/Qwen3-VL-30B-A3B-Instruct
• SFT: LoRA rank 64 on part-by-part sketch continuation data; the released checkpoint's RL run started from SFT step 5400.
• RL: multi-turn GRPO with dense step-wise advantages and a DreamSim visual reward; this is the checkpoint at RL step 1000.
• Trained with Tinker. Note that Tinker has since deprecated Qwen/Qwen3-VL-30B-A3B-Instruct (its replacement there is Qwen/Qwen3.6-35B-A3B); this affects re-training on Tinker only — local inference with this repo is unaffected.

Related resources

• Paper: arXiv:2603.19500
• Dataset: duxiaodan/ControlSketch-Part
• Code: GitHub

License

Apache 2.0, following the base model.

Citation

@article{du2026sketch,
  title   = {Teaching an Agent to Sketch One Part at a Time},
  author  = {Du, Xiaodan and Xu, Ruize and Yunis, David and Vinker, Yael and Shakhnarovich, Greg},
  journal = {arXiv preprint arXiv:2603.19500},
  year    = {2026}
}