ericrcwu/LUT_SLM_sft_adapters

LUT-SLM — Stage-2 Generator Adapters (QLoRA over Qwen2.5-VL-3B)

QLoRA adapters for the Stage-2 generator of the LUT-SLM project: a small vision-language model that turns (source image + natural-language photo-editing instruction) into a single global color Look-Up Table (LUT). Given "make it warmer and lift the shadows" the model emits the tokens of a 17³ .cube LUT that bakes in exactly that look; given a request a single global LUT physically cannot satisfy (e.g. "remove the person on the left") it emits <unsupported> and refuses.

These adapters are trained on the companion dataset ericrcwu/LUT_SLM (see that card for the full data story). The Stage-1 request router lives in ericrcwu/LUT_SLM_interpreter.

Status — research artifacts, work in progress. These are smoke-scale / bilevel-search run outputs, not a finalized release. Treat them as reproducible checkpoints from the collapse-fix and two-stage experiments.

What's in this repo

Each subfolder is a self-contained PEFT adapter (adapter weights + tokenizer + chat template + adapter_manifest.json), except distill_r1_distilled_corpus/, which holds a distilled data corpus rather than weights.

Subfolder	What it is
p6_twostage_d0f9c744_smokefull/	Deployed generator. P6 two-stage run adapter used by the webapp / Modal deploy (deploy/modal_app.py). mean train loss ≈ 1.677, 182 steps, lr 2e-4.
bl_63cd1bf7_smokefull/	One-stage full-run winner from the bilevel-over-SFT search. mean train loss ≈ 1.747, 162 steps, lr 3e-4.
bl_a0ccbcff_smokefull/	Bilevel baseline adapter (full smoke run).
bl_a0ccbcff_smoke600/	Bilevel baseline adapter (600-example smoke run).
distill_r1_smokefull/	Distillation round-1 adapter.
distill_r1_distilled_corpus/	Distilled corpus (active_rows.jsonl, active_manifest.json, harvest_cache.jsonl) — data, not weights.

Shared architecture & training recipe

All adapters share the same shape (per adapter_manifest.json):

• Base model: Qwen/Qwen2.5-VL-3B-Instruct, with the output embedding resized to 151,924 tokens — the base vocab plus 259 special LUT tokens (<lut_bos>, <lut_eos>, <unsupported>, <lut_000>…<lut_255>). Embeddings are tied.
• LoRA: r = 16, alpha = 32, dropout = 0.05, targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Quantization: 4-bit QLoRA — nf4, double-quant, bf16 compute dtype.
• Optim: effective batch size 32 (per-device 1 × grad-accum 32), cosine schedule, 3% warmup, grad-checkpointing, 2 epochs, seed 0.
• Frozen VQ tokenizer: the 64 LUT code tokens decode via tokenizer_version = vq_v2_srgbres_17to4_cb256_t64… (encoder 17³ → 4³ latent → 64 codes over a 256-entry codebook; decoder → a residual LUT added to the sRGB identity grid → .cube). The tokenizer artifacts themselves ship with the LUT_SLM corpus shards.

Output grammar: supported → <lut_bos> <lut_###> ×64 <lut_eos>; unsupported → <unsupported>.

How to load

The adapter targets a vocab-resized base, so you must resize the base embeddings to 151,924 and add the 259 special tokens before attaching the adapter (the adapter_config.json base_model field points at a local models/base_resized, i.e. the resized base — not a Hub repo).

from huggingface_hub import snapshot_download
d = snapshot_download("ericrcwu/LUT_SLM_sft_adapters",
                      allow_patterns=["p6_twostage_d0f9c744_smokefull/*"])
# 1) load Qwen/Qwen2.5-VL-3B-Instruct, 2) add the 259 special tokens + resize embeddings to 151924,
# 3) PeftModel.from_pretrained(base, f"{d}/p6_twostage_d0f9c744_smokefull").
# See notebooks/colab_lut_slm_inference.ipynb in the source repo for a runnable end-to-end example
# (vocab is reconstructed in memory, LUT codes decoded with the frozen tokenizer, image rendered).

Licensing & provenance

license: other. The base model is governed by its own Qwen license; these adapters are derived from the mixed-provenance LUT_SLM corpus, several sources of which are personal-use / non-redistribution (see that dataset's licensing section). This repository makes no license claim over the underlying LUTs or images. Research use; verify each source family's original terms before any redistribution or commercial use.