diffusiondb-sd15-lora

A LoRA adapter for Stable Diffusion 1.5, fine-tuned on a cleaned, safety-filtered slice of DiffusionDB.

Status: not yet trained. This repository was created ahead of the first run. Checkpoints and sample renders appear here as training proceeds.

What this is for

DiffusionDB is itself Stable Diffusion 1.x output, collected from the official Stable Diffusion Discord. Fine-tuning SD 1.5 on it therefore shifts the model toward the DiffusionDB aesthetic — the keyword-heavy artstation / intricate / octane render idiom that its users prompted with. It does not push image quality past what SD 1.5 already does, and it is not intended to. Judge it on style adherence, not on "is it better than the base model".

Training data

whosouravsharma/text-to-image-diffusiondb-2M at revision v2-clean — 14,598 images from part_id 1–20.

split	examples
train	13,598
validation	1,000

The validation split is separated by normalized prompt group, not by row. DiffusionDB users re-roll the same prompt at many seeds, so a random row split leaks: in an earlier revision, 21% of validation prompts had already been seen in training. Every image sharing a prompt now lands on the same side.

Filters applied upstream: prompts under 4 words dropped, short side ≥ 384 px and area ≥ 262,144 px, image_nsfw and prompt_nsfw below 0.2, at most 2 images per normalized prompt, exact SHA-256 duplicates removed.

Configuration

base	runwayml/stable-diffusion-v1-5
adapter	LoRA rank 32, alpha 32, on UNet to_q/to_k/to_v/to_out.0
text encoder	frozen
resolution	512×512, centre crop
VAE	stabilityai/sd-vae-ft-mse (latents cached ahead of training)
effective batch	32 (8 × 4 grad accumulation)
optimizer	AdamW, lr 1e-4, cosine schedule, 500 warmup
caption dropout	10%, for classifier-free guidance
precision	fp16

Each checkpoint carries a state.json recording the exact values it was trained with, so the table above can be checked rather than trusted.

Repository layout

checkpoints/
  checkpoint-<step>/
    pytorch_lora_weights.safetensors   the adapter
    optimizer.pt                       optimizer + scaler state, for resuming
    state.json                         step, epoch, hyperparameters
training/                              the scripts that produced all of this
samples/
  base/                              vanilla SD 1.5, the comparison baseline
  checkpoint-<step>/
    grid.jpg                         contact sheet, all eval prompts
    000.png … 049.png                individual renders
    prompts.json                     prompt list, seed, steps, guidance

Evaluation

50 prompts held out from training entirely (drawn from validation groups) are rendered at every checkpoint with a fixed per-prompt seed, so differences between contact sheets come from the weights rather than from noise. samples/base/ is vanilla SD 1.5 on the same prompts.

Validation loss is logged per epoch, but on a diffusion fine-tune it tracks output quality only loosely — it is there to catch divergence, not to rank checkpoints.

Usage

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")
pipe.load_lora_weights(
    "whosouravsharma/diffusiondb-sd15-lora", subfolder="checkpoints/checkpoint-4240"
)

image = pipe(
    "a steampunk owl inside a glass jar, intricate detail",
    num_inference_steps=30, guidance_scale=7.5,
).images[0]

Limitations

• Trained on ~13.6k images — fine-tuning scale, not from-scratch scale.
• Source images are SD 1.x generations, so artifacts of that model are reproduced along with its style.
• 512×512 centre crop discards roughly 16% of the source image area; ~10% of images lose more than 40% of their frame.
• NSFW filtering relies on DiffusionDB's own classifier scores at a 0.2 threshold. That classifier is noisy, so the training set is filtered, not guaranteed clean.
• Prompts carry heavy style boilerplate (~22% mention artstation), which the adapter will have learned as part of the aesthetic.

License & attribution

Adapter released under CreativeML OpenRAIL-M, matching the SD 1.5 base model. Training data derives from poloclub/diffusiondb, CC0-1.0. Wang et al., 2022, arXiv:2210.14896.