art-lora-flux1-dev

Style LoRA for FLUX.1-dev. Trained on 19 curated painterly character-design references.

Trigger: sk3tchref

Hand-painted character concept art — broad flat brush shapes, hard-edged colour blocking, minimal blending, muted olive / charcoal / cream / tan palette, flat off-white background with a thin shadow smear, stylized elongated proportions.

---

Repo contents

Path	What it is
sk3tchref.safetensors	Final weights (step 2000) — use this one
sk3tchref-step000005/1000/1500/2000.safetensors	Intermediate checkpoints
workflow.json	ComfyUI workflow — drag onto the canvas
workflow_api.json	Same graph in API format, for POST /prompt
dataset/	All 19 training images and their caption files
samples/	Outputs per checkpoint + comparison grids

Style LoRAs overfit, so the best checkpoint is not always the last. All of them are here so you can compare — see samples/_grids/.

---

1. Download the models

Everything below goes into your ComfyUI models/ tree. Total ~34 GB.

cd /workspace/ComfyUI      # adjust to your ComfyUI root
mkdir -p models/diffusion_models models/text_encoders models/vae models/loras

Transformer — FLUX.1-dev, 23.8 GB

The official repo black-forest-labs/FLUX.1-dev is gated: you must accept the licence on the model page and use a read token. Same for FLUX.1-schnell.

# Option A — official, gated. Accept the licence first at
#   https://huggingface.co/black-forest-labs/FLUX.1-dev
# then create a READ token at https://huggingface.co/settings/tokens
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx

wget --header="Authorization: Bearer $HF_TOKEN" \
  -O models/diffusion_models/flux1-dev.safetensors \
  "https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors"

# Option B — Comfy-Org mirror. Identical weights, no token needed at time of writing.
wget -O models/diffusion_models/flux1-dev.safetensors \
  "https://huggingface.co/Comfy-Org/flux1-dev/resolve/main/flux1-dev.safetensors"

Option B is what this LoRA was trained against. Mirrors can be gated or removed without notice — if Option B starts returning 401, fall back to Option A.

Text encoders — 10 GB

wget -O models/text_encoders/t5xxl_fp16.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp16.safetensors"

wget -O models/text_encoders/clip_l.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors"

VAE — 335 MB

# ungated mirror (black-forest-labs/FLUX.1-schnell is gated and returns 401)
wget -O models/vae/ae.safetensors \
  "https://huggingface.co/ffxvs/vae-flux/resolve/main/ae.safetensors"

This LoRA

wget -O models/loras/sk3tchref.safetensors \
  "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/sk3tchref.safetensors"

# intermediate checkpoints, if you want to compare
for S in 00000500 00001000 00001500 00002000; do
  wget -O "models/loras/sk3tchref-step$S.safetensors" \
    "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/sk3tchref-step$S.safetensors"
done

If the repo is private, add --header="Authorization: Bearer $HF_TOKEN" to those too.

Verify

A truncated download does not error — it fails later at load time with 'NoneType' object has no attribute 'Params'. Check the sizes:

ls -l models/diffusion_models/flux1-dev.safetensors   # 23802932552
ls -l models/text_encoders/t5xxl_fp16.safetensors     #  9787841024
ls -l models/text_encoders/clip_l.safetensors         #   246144152
ls -l models/vae/ae.safetensors                       #   335304388

---

2. Run it

wget -O workflow.json \
  "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/workflow.json"

Drag workflow.json onto the ComfyUI canvas. It wires up:

UNETLoader (flux1-dev, fp8_e4m3fn)
  └─ LoraLoaderModelOnly (sk3tchref.safetensors, strength 1.0)
       └─ KSampler ── VAEDecode ── SaveImage
DualCLIPLoader (clip_l + t5xxl_fp16, type "flux")
  └─ CLIPTextEncode ─┬─ FluxGuidance 3.0 ──→ positive
                     └─ ConditioningZeroOut ─→ negative

workflow_api.json is the same graph for POST /prompt.

Settings that matter

cfg must stay at 1.0. Flux dev is CFG-distilled — raising it destroys the image, and negative prompts do nothing, which is why the negative branch is just a zeroed-out copy of the positive. FluxGuidance (~3.0) is the real prompt-adherence knob.

Sampler euler, scheduler beta, 28 steps, 832×1216.

Prompt template

sk3tchref, a full body character illustration of <SUBJECT>, hand-painted character
concept art, broad flat brush shapes, hard-edged colour blocking, muted desaturated
palette, flat off-white background, thin grey shadow smear under the feet

The trigger must come first.

Strength guide

Strength	Result
0.6	Too weak — the base model's smooth prior shows through
0.8	Works
1.0	Recommended — strongest read, no subject bleed
1.1+	Training subjects and poses start reappearing

See samples/final/sweep.jpg.

---

3. How it was trained

kohya sd-scripts (sd3 branch), flux_train_network.py, on an RTX 5090 32 GB. ~1.75 s/it, ~1 hour for 2000 steps.

accelerate launch --num_processes 1 --mixed_precision bf16 \
  --num_cpu_threads_per_process 8 flux_train_network.py \
  --pretrained_model_name_or_path $M/diffusion_models/flux1-dev.safetensors \
  --clip_l $M/text_encoders/clip_l.safetensors \
  --t5xxl $M/text_encoders/t5xxl_fp16.safetensors \
  --ae $M/vae/ae.safetensors \
  --dataset_config dataset.toml \
  --output_dir ./out --output_name sk3tchref \
  --save_model_as safetensors --save_precision bf16 \
  --network_module networks.lora_flux --network_dim 24 --network_alpha 24 \
  --network_train_unet_only \
  --optimizer_type adamw8bit --learning_rate 1e-4 \
  --lr_scheduler constant_with_warmup --lr_warmup_steps 40 \
  --max_train_steps 2000 --save_every_n_steps 500 \
  --gradient_checkpointing --mixed_precision bf16 --fp8_base --sdpa --highvram \
  --cache_latents_to_disk --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
  --max_data_loader_n_workers 2 --persistent_data_loader_workers --seed 42 \
  --timestep_sampling shift --discrete_flow_shift 3.1582 \
  --model_prediction_type raw --guidance_scale 1.0 --loss_type l2

Dataset config: aspect-ratio bucketing at 1024 base, min_bucket_reso 512, max_bucket_reso 1536, keep_tokens = 1, shuffle_caption = false.

---

4. The captioning method

This is the part that decides whether a style LoRA works.

Captions describe content only — who the character is, what they wear, what they do:

sk3tchref, two views, front and back, of a heavy-set older man with a grey beard in an
olive green jacket, cream shirt and dark green trousers, holding a bottle

Note what is absent: no "painterly", no "flat colour blocking", no "muted palette", no "white background". Everything constant across the set and missing from the captions collapses onto the single token they all share — sk3tchref.

Write style words into the captions and the style scatters across those common words instead, producing a LoRA that is weak, hard to trigger, and unresponsive to strength. keep_tokens = 1 pins the trigger at the front so shuffling never displaces it.

Every caption in dataset/ was written by looking at the image.

---

Why a LoRA and not --sref

Reference-image conditioning was tested first and does not work for style transfer:

Approach	Result
Redux / IPAdapter	Encodes the reference into content tokens. Below ~0.2 strength nothing transfers; above it the reference's subject appears — its hat, its props, its character. No setting yields brushwork without subject bleed.
Flux.2 Klein / Kontext edit-refs	Trained to preserve reference content; "style only" fights the objective.
Prompt engineering alone	Yields Flux's own house style. Sweeping guidance moves composition, not paint.

Conditioning transfers content. Prompting transfers the base model's prior. Only weight updates transfer style.