---

license: apache-2.0 base_model: Qwen/Qwen3.5-0.8B tags: - layout-planning - prompt-upsampling - distillation - bbox

pocket-layout-planner-0.8B

A Qwen3.5-0.8B fine-tune that takes only a short image request (no system prompt, no thinking) and returns, on one line, a detailed caption citing elements as <id> followed by a JSON box table placing each element — [y_min, x_min, y_max, x_max] on a 0–1000 grid, top-left origin. Distilled from the t2i_bbox mode of black-forest-labs/flux-3-image-klein-pup (Apache-2.0, credited as teacher).

The box-table format is the format FLUX.3 Image Klein consumes as its "DiT prompt format"; boxes are normalised, so the same plan renders at any size.

Evaluation (300-request stratified hold-out + 100-prompt image comparison)

Text-level — all 301 hold-out requests, greedy, nothing dropped:

Metric	0.8B SFT (this model)	2B SFT	2B untuned + t2i_bbox prompt
parse + fully-cited rate	95.0%	92.7%	1.0%
valid-box rate	96.4%	95.7%	10.6%
quoted text verbatim	55/55	55/55	55/55
mean matched-row IoU to teacher	0.470	0.529	0.133
mean output tokens	618	646	810

CPU latency (GGUF Q8_0, llama.cpp, 8 vCPU): 30.44 ± 1.25 tok/s (tg128) → a mean 620-token plan in ≈ 20 s on CPU.

Image-level — 100-prompt comparison (60 GenEval-subset + 40 spec requests), rendered with FLUX.3 Image Klein fp8r at 512², 1 seed, scored with a clearly-labeled "GenEval-style (OWLv2)" re-implementation (not official GenEval) plus detector-based object counts:

Arm	GenEval-style (60)	Spec-request object counts (40)
raw request	58/60	29/40
pup plain t2i	60/60	30/40
pup t2i_bbox (teacher)	60/60	28/40
this 0.8B	53/60	26/40
2B SFT	58/60	26/40

Finding, stated plainly: no layout advantage materialized. Rendering from structured box tables — whether from the 9B teacher or these students — does not beat the raw request or plain upsampling on object counts, and Klein distilled at 512² is already near-saturated on object presence for every arm. These models are therefore best understood as pocket prompt upsamplers that happen to emit a structured, parseable layout record (96–97% reliable), not as layout planners that improve adherence. The planned DPO stage was dropped as moot. Full data: results/ in the dataset repo, including the contact sheet.

Training recipe

• Teacher labels: 7,842 taxonomy-stratified requests (14% non-English, 27% with TARGET CANVAS, 40% with structured specs) through the pup's t2i_bbox mode on vLLM (greedy, official sampling params); 97.0% passed the hard filter → 7,575 clean rows. Data: ysharma/klein-layout-plans.
• SFT: TRL SFTTrainer, conversational prompt-completion (loss on assistant tokens only), 2 epochs (10.1M tokens), LR 1e-5 cosine, batch 16, max_length 1536, bf16, vision tower frozen, on an A10G (2h11m). Padded batching, not packed: Qwen3.5's linear-attention layers require the CUDA-only causal_conv1d kernel for TRL's packing — decided by evidence.
• Overfit gate before training: 32 rows → 100% parse, matched IoU 1.0 vs teacher.

Executed snippet

Executed end-to-end during evaluation (the image-comparison table above rendered 193 plans through exactly this path):

import json, torch
from transformers import AutoTokenizer, AutoModelForImageTextToText

model_id = "ysharma/pocket-layout-planner-0.8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16).eval()

request = "four apples and three oranges arranged in a wooden crate"
text = tok.apply_chat_template([{"role": "user", "content": request}],
                               tokenize=False, add_generation_prompt=True, enable_thinking=False)
ids = tok(text, return_tensors="pt")
plan = tok.decode(model.generate(**ids, max_new_tokens=1500, do_sample=False)[0, ids["input_ids"].shape[1]:],
                  skip_special_tokens=True).strip()
print(plan)   # caption with <id> citations, then [{"id": ..., "bbox": [y,x,y,x], "desc": ...}, ...]

# render with FLUX.3 Image Klein (FLUX Kommunity Licence weights):
from diffusers import ModularPipeline
pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein")
pipe.load_components(dtype=torch.bfloat16)
image = pipe(prompt=plan, num_inference_steps=16,
             generator=torch.Generator(device="cpu").manual_seed(0),
             output="images", height=512, width=512)[0]
image.save("out.png")

Limitations

• Layout does not improve Klein distilled renders (see the image-level table) — use these models as compact upsamplers, not as adherence-boosting planners. Whether full-size Klein or the base (non-distilled) variant benefits from box tables was not measured.
• Object-count satisfaction of renders trails the raw request slightly (26/40 vs 29/40); the teacher's own layout plans show the same pattern (28/40 vs 30/40 for its plain mode).
• Output is English; requests in any language (14% of training requests were non-English).
• Spec-count satisfaction of its own plans is ~30% of verifiable spec rows — the checker's vocabulary limits are documented in the dataset repo; do not read that as a 70% error rate.
• The image comparison is 100 prompts, 1 seed, presence-count scoring — far below the official 553 × 4-seed GenEval protocol.

Licence

Apache-2.0 (inherited from Qwen3.5-0.8B). Teacher credit: black-forest-labs/flux-3-image-klein-pup (Apache-2.0). Rendering weights (FLUX.3 Image Klein) are under the FLUX Kommunity Licence — this repo contains no Klein weights and is not affiliated with or endorsed by Black Forest Labs.