sorryhyun/moss-image-colorize

MOSS-Image colorize LoRA (i2i probe v1)

Two of the eight held-out pages, seed 1, from the eval below. In reading order (left: top to bottom; right: 2×2): the B&W page (the input), the colour target, this LoRA with the colour caption, and this LoRA with the page only.

• Left (touhou). Caption: touhou, purple eyes, purple hair, purple eyes, black hairband, blue shirt, green hair, yellow shirt, black hat.
• Right (blue archive). Caption: blue archive, black pantyhose, blue necktie, grey hair, purple eyes, white shirt, white skirt.

The line work and layout follow the page in both modes. With a caption, the hues follow it (left: purple and green hair). With the page only, the LoRA picks its own hues (left: blue-green coat, brown hair).

A LoRA for MOSS-Image Base that colorizes a black-and-white manga page. The page goes in twice: through Qwen3.5's vision tower inside the text encoder, and as a clean VAE latent in the DiT sequence. The output keeps the page's line work, faces and lettering.

Status: research probe. One 1-hour run, 1000 training pairs, 8 held-out pages, and one seed per page. Base was not trained for image conditioning. This LoRA teaches it to read a reference.

Workflow

Install the custom node in here

Drag and drop the following image to comfyui.

How it works

MOSS-Image has no image input. Its 4D RoPE (r, f, h, w) has an r axis that its training always set to 0. This LoRA was trained with:

• Text encoder input. The page through the vision tower, then the colour caption, in one user turn: <|im_start|>user\n<|vision_start|>[page]<|vision_end|>{caption}<|im_end|>\n<|im_start|>assistant\n<think>\n. At the target's pixel count, that is ~1000 image tokens at 1 MP. The vision tower alone brings back content but not line placement: one token per 32×32 px.
• VAE reference. The page at half size per side (1/4 area) goes into the DiT sequence as [image | ref | text].
  • The ref tokens get no noise and share the step's timestep modulation.
  • On RoPE they sit at r = 1, with their (h, w) stretched onto the target grid, so each ref token lines up with the region it covers.
  • The ref goes into every CFG branch. The head and the loss read the image tokens only.

Files

File	What
moss_colorize_ref_v1.safetensors	The LoRA: rank 32, alpha 128, on proj.{q,k,v,o,ffn_up_gate,ffn_down} of all 42 blocks, bf16, 149 MB. Plain lora_unet_layers_* keys; channel scaling is already folded into the stored weights
workflows/moss-i2i-colorize.json	ComfyUI example workflow

ComfyUI

It needs two custom nodes:

• moss-image (OpenMOSS): the MODEL / CLIP / VAE loaders.
• ComfyUI-MOSS-i2i: three nodes for this LoRA.
  • MOSS i2i LoRA Loader.
  • MOSS i2i Text Encode (page + text): the vision-tower encode above.
  • MOSS i2i Reference: the VAE ref, plus an empty latent at the target size.
  • The node is not published yet.

Wiring (workflows/moss-i2i-colorize.json):

Load Image (page) ──┬─────────────────────────────┐
                    │                             │
MOSS-Image HF Loader ─ MODEL ─ i2i LoRA Loader ─ i2i Reference ─ ModelSamplingSD3(5) ─ SLG(single 22) ─┐
                     ─ CLIP ─┬ i2i Text Encode (page, colour tags) ── positive ─────────────────────────┤
                             └ CLIP Text Encode ("") ──────────────── negative ─────────────────────────┤ KSampler
                     ─ VAE ── i2i Reference ── LATENT ──────────────────────────────────────────────────┘

• Positive. The page plus colour tags: copyright first, then colours, e.g. touhou, purple eyes, purple hair, black hairband, blue shirt. An empty text gives the page-only mode. Its colours are plausible, but nothing pins the hue.
• Negative. A plain text encode of "".
• Resolution. Use the same value (1024) on Text Encode and Reference. The page is fit to ~1 MP at its native aspect, with sides multiples of 16.
• Sampler. CFG 3, with block 22 skipped on the uncond branch, and shift 5. The workflow uses ER-SDE at 30 steps, the setting the eval below used. The MOSS Spectrum KSampler (ComfyUI-Spectrum-KSampler) works too.

Training

Data	1000 pairs from a colorize set: a synthetic mangafied page → its colour original
Caption	Colour tags only: the copyright and comic tags are kept, all others dropped. 30 % of steps use the page with no caption
Steps	1600 (2 epochs), lr 1e-4, 5 % warmup, max grad norm 1.0
Adapter	LoRA r32 / α128, weight-SVD down init, channel scaling α 0.5, DC isolation
σ sampling	logit-normal μ −1.09, s 0.95, cutoff 0.95, 1 % uniform
Cost	71 min at 2.67 s/step on one consumer GPU (36 blocks swapped)

Eval (8 held-out pages, seed 1)

Base + LoRA, ER-SDE + Spectrum, 30 steps, CFG 3. Base gets no ref (it never saw one), so the fair comparison is the vision-tower-only LoRA (v1, same recipe, no VAE ref) against this one.

Limits

• Trained and evaluated on Base. SFT is untested.
• Ref size 1/2 per side is the only one tried. ref_scale is not yet stamped in the LoRA metadata, so the Reference node defaults to 2.