metadata

license: mit
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
  - stable-diffusion
  - stable-diffusion-diffusers
  - text-to-image
  - diffusers
  - lora
inference: true

sdxl-wrong-lora

A LoRA for SDXL 1.0 Base which improves output image quality after loading it and using wrong as a negative prompt during inference.

Benefits of using this LoRA:

Higher color saturation and vibrance
Higher detail in textures/fabrics
Higher sharpness for blurry/background objects
Better at anatomically-correct hands
Less likely to have random artifacts
Appears to allow the model to follow the input prompt with a more expected behavior

Usage

The LoRA can be loaded using load_lora_weights like any other LoRA in diffusers:

import torch
from diffusers import DiffusionPipeline, AutoencoderKL

vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix",
    torch_dtype=torch.float16
)
base = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    vae=vae,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
)

base.load_lora_weights("minimaxir/sdxl-wrong-lora")

_ = base.to("cuda")

During inference, use wrong as the negative prompt.

Examples

Left is the base model output (no LoRA) + refiner, right is base + LoRA and refiner. The generations use the same seed.

Methodology

The methodology and motivation for creating this LoRA is similar to my wrong SD 2.0 textual inversion embedding by training on a balanced variety of undesirable outputs, except trained as a LoRA since textual inversion on SDXL is complicated. The base images were generated from SDXL itself, with some prompt weighting to emphasize undesirable attributes for test images.

Notes

The intuitive way to think about how this LoRA works
It's possible to use not wrong in the prompt itself but in testing it has no effect.
You can use other negative prompts in conjunction with the wrong prompt but you may want to weight them appropriately.