README.md

---
license: other
base_model: "stabilityai/stable-diffusion-3-medium-diffusers"
tags:
  - sd3
  - sd3-diffusers
  - text-to-image
  - diffusers
  - simpletuner
  - not-for-all-audiences
  - lora
  - template:sd-lora
  - lycoris
inference: true
widget:
- text: 'unconditional (blank prompt)'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_0_0.png
- text: 'A swift and agile elven archer perched in a tree, nocking an arrow.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_1_0.png
- text: 'A cyberpunk hunter in neon-lit city alleys, armed with a high-tech rifle.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_2_0.png
- text: 'A mighty fantasy knight in gleaming armor, wielding a sword and shield.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_3_0.png
- text: 'A space pirate captain standing on the bridge of a starship, ready for adventure.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_4_0.png
- text: 'A powerful demonic sorcerer casting a spell in a dark, mysterious chamber.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_5_0.png
- text: 'A friendly robotic assistant with a sleek design, helping a player navigate a game.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_6_0.png
- text: 'A stealthy ninja warrior crouching in the shadows, ready to strike.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_7_0.png
- text: 'A group of survivors in a post-apocalyptic world, fending off a zombie horde.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_8_0.png
- text: 'A brave dragon tamer soaring through the sky on the back of a majestic dragon.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_9_0.png
- text: 'A wise medieval wizard in a towering castle, studying ancient tomes of magic.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_10_0.png
- text: 'Sun Wukong, the Monkey King from ''Black Myth: Wukong'', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong''s fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_11_0.png
- text: 'Sun Wukong, the Monkey King from ''Black Myth: Wukong'', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong''s fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.'
  parameters:
    negative_prompt: 'blurry, cropped, ugly'
  output:
    url: ./assets/image_12_0.png
---

# simpletuner-lora

This is a LyCORIS adapter derived from [stabilityai/stable-diffusion-3-medium-diffusers](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers).


The main validation prompt used during training was:
```
Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.
```


## Validation settings
- CFG: `3.0`
- CFG Rescale: `0.0`
- Steps: `20`
- Sampler: `FlowMatchEulerDiscreteScheduler`
- Seed: `42`
- Resolution: `1024x1024`
- Skip-layer guidance: 

Note: The validation settings are not necessarily the same as the [training settings](#training-settings).

You can find some example images in the following gallery:


<Gallery />

The text encoder **was not** trained.
You may reuse the base model text encoder for inference.


## Training settings

- Training epochs: 10
- Training steps: 10000
- Learning rate: 0.0001
  - Learning rate schedule: polynomial
  - Warmup steps: 100
- Max grad norm: 0.01
- Effective batch size: 1
  - Micro-batch size: 1
  - Gradient accumulation steps: 1
  - Number of GPUs: 1
- Gradient checkpointing: True
- Prediction type: flow-matching (extra parameters=['shift=3'])
- Optimizer: adamw_bf16
- Trainable parameter precision: Pure BF16
- Caption dropout probability: 10.0%


### LyCORIS Config:
```json
{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 16,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 16
            },
            "FeedForward": {
                "factor": 8
            }
        }
    }
}
```

## Datasets

### wikiart_sargent
- Repeats: 0
- Total number of images: 920
- Total number of aspect buckets: 4
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No


## Inference


```python
import torch
from diffusers import DiffusionPipeline
from lycoris import create_lycoris_from_weights


def download_adapter(repo_id: str):
    import os
    from huggingface_hub import hf_hub_download
    adapter_filename = "pytorch_lora_weights.safetensors"
    cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models'))
    cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_")
    path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path)
    path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename)
    os.makedirs(path_to_adapter, exist_ok=True)
    hf_hub_download(
        repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter
    )

    return path_to_adapter_file
    
model_id = 'stabilityai/stable-diffusion-3-medium-diffusers'
adapter_repo_id = 'jimchoi/simpletuner-lora'
adapter_filename = 'pytorch_lora_weights.safetensors'
adapter_file_path = download_adapter(repo_id=adapter_repo_id)
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
lora_scale = 1.0
wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer)
wrapper.merge_to()

prompt = "Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility."
negative_prompt = 'blurry, cropped, ugly'

## Optional: quantise the model to save on vram.
## Note: The model was quantised during training, and so it is recommended to do the same during inference time.
from optimum.quanto import quantize, freeze, qint8
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
    
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
    width=1024,
    height=1024,
    guidance_scale=3.0,
).images[0]
image.save("output.png", format="PNG")
```