simpletuner-lora

This is a LyCORIS adapter derived from stabilityai/stable-diffusion-3-medium-diffusers.

The main validation prompt used during training was:

Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.

Validation settings

CFG: 3.0
CFG Rescale: 0.0
Steps: 20
Sampler: FlowMatchEulerDiscreteScheduler
Seed: 42
Resolution: 1024x1024
Skip-layer guidance:

Note: The validation settings are not necessarily the same as the training settings.

You can find some example images in the following gallery:

Prompt
unconditional (blank prompt)

Negative Prompt
blurry, cropped, ugly

Prompt
A swift and agile elven archer perched in a tree, nocking an arrow.

Negative Prompt
blurry, cropped, ugly

Prompt
A cyberpunk hunter in neon-lit city alleys, armed with a high-tech rifle.

Negative Prompt
blurry, cropped, ugly

Prompt
A mighty fantasy knight in gleaming armor, wielding a sword and shield.

Negative Prompt
blurry, cropped, ugly

Prompt
A space pirate captain standing on the bridge of a starship, ready for adventure.

Negative Prompt
blurry, cropped, ugly

Prompt
A powerful demonic sorcerer casting a spell in a dark, mysterious chamber.

Negative Prompt
blurry, cropped, ugly

Prompt
A friendly robotic assistant with a sleek design, helping a player navigate a game.

Negative Prompt
blurry, cropped, ugly

Prompt
A stealthy ninja warrior crouching in the shadows, ready to strike.

Negative Prompt
blurry, cropped, ugly

Prompt
A group of survivors in a post-apocalyptic world, fending off a zombie horde.

Negative Prompt
blurry, cropped, ugly

Prompt
A brave dragon tamer soaring through the sky on the back of a majestic dragon.

Negative Prompt
blurry, cropped, ugly

Prompt
A wise medieval wizard in a towering castle, studying ancient tomes of magic.

Negative Prompt
blurry, cropped, ugly

Prompt
Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.

Negative Prompt
blurry, cropped, ugly

The text encoder was not trained. You may reuse the base model text encoder for inference.

Training settings

Training epochs: 10
Training steps: 10000
Learning rate: 0.0001
- Learning rate schedule: polynomial
- Warmup steps: 100
Max grad norm: 0.01
Effective batch size: 1
- Micro-batch size: 1
- Gradient accumulation steps: 1
- Number of GPUs: 1
Gradient checkpointing: True
Prediction type: flow-matching (extra parameters=['shift=3'])
Optimizer: adamw_bf16
Trainable parameter precision: Pure BF16
Caption dropout probability: 10.0%

LyCORIS Config:

{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 16,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 16
            },
            "FeedForward": {
                "factor": 8
            }
        }
    }
}

Datasets

wikiart_sargent

Repeats: 0
Total number of images: 920
Total number of aspect buckets: 4
Resolution: 1.048576 megapixels
Cropped: False
Crop style: None
Crop aspect: None
Used for regularisation data: No

Inference

import torch
from diffusers import DiffusionPipeline
from lycoris import create_lycoris_from_weights


def download_adapter(repo_id: str):
    import os
    from huggingface_hub import hf_hub_download
    adapter_filename = "pytorch_lora_weights.safetensors"
    cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models'))
    cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_")
    path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path)
    path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename)
    os.makedirs(path_to_adapter, exist_ok=True)
    hf_hub_download(
        repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter
    )

    return path_to_adapter_file
    
model_id = 'stabilityai/stable-diffusion-3-medium-diffusers'
adapter_repo_id = 'jimchoi/simpletuner-lora'
adapter_filename = 'pytorch_lora_weights.safetensors'
adapter_file_path = download_adapter(repo_id=adapter_repo_id)
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
lora_scale = 1.0
wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer)
wrapper.merge_to()

prompt = "Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility."
negative_prompt = 'blurry, cropped, ugly'

## Optional: quantise the model to save on vram.
## Note: The model was quantised during training, and so it is recommended to do the same during inference time.
from optimum.quanto import quantize, freeze, qint8
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
    
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
    width=1024,
    height=1024,
    guidance_scale=3.0,
).images[0]
image.save("output.png", format="PNG")

jimchoi
/

simpletuner-lora