README.md

metadata

license: other
base_model: black-forest-labs/FLUX.1-dev
tags:
  - flux
  - flux-diffusers
  - text-to-image
  - diffusers
  - simpletuner
  - safe-for-work
  - lora
  - template:sd-lora
  - lycoris
inference: true
widget:
  - text: unconditional (blank prompt)
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_0_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, Three stag beetles on
      oak bark, with one near green leaves at the top, another climbing
      vertically in the middle, and a third at the base amid fallen leaves and
      moss.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_1_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, Four large moths
      around green leaves, one cream-colored, two brown with circular wing
      patterns, and one white moth in flight, with a pale caterpillar climbing
      on a leaf above.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_2_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, A golden hamster sits
      upright on desert sand, its cheek pouches full of seeds. Three small
      scarab beetles move across the sand nearby, while a scorpion rests in the
      lower right corner.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_3_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, A Range Rover in an
      African savanna setting, with two rhinoceros beetles on its front tire.
      Three dung beetles roll balls past its tracks in the dirt, while acacia
      trees stand in the background.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_4_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, A glass Coca-Cola
      bottle lying sideways on brown leaves and soil. A line of black ants
      traverses its red label, two iridescent beetles explore the metal cap, and
      a pale moth rests on the glass neck.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_5_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, Black over-ear
      headphones on a wooden table. Three small beetles crawl along the ear
      cushions, while a spider hangs between the headband adjusters, its web
      gleaming in the light.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_6_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, A white athletic shoe
      on packed earth. Carpenter ants march through its eyelets, a beetle rests
      under the loosened tongue, and a cricket perches on the heel.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_7_0.png
  - text: >-
      In the style of an Alfred Edmund Brehm illustration, Three wooden pencils
      lying across a blank paper sheet. A praying mantis stands on one pencil
      tip, while two ladybugs explore graphite shavings scattered below.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_8_0.png

AlbertBierstadt-Flux-LoKr

This is a LyCORIS adapter derived from black-forest-labs/FLUX.1-dev.

No validation prompt was used during training.

None

Validation settings

• CFG: 3.0
• CFG Rescale: 0.0
• Steps: 20
• Sampler: FlowMatchEulerDiscreteScheduler
• Seed: 42
• Resolution: 1024x1280
• Skip-layer guidance:

Note: The validation settings are not necessarily the same as the training settings.

You can find some example images in the following gallery:

Prompt

unconditional (blank prompt)

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, Three stag beetles on oak bark, with one near green leaves at the top, another climbing vertically in the middle, and a third at the base amid fallen leaves and moss.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, Four large moths around green leaves, one cream-colored, two brown with circular wing patterns, and one white moth in flight, with a pale caterpillar climbing on a leaf above.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, A golden hamster sits upright on desert sand, its cheek pouches full of seeds. Three small scarab beetles move across the sand nearby, while a scorpion rests in the lower right corner.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, A Range Rover in an African savanna setting, with two rhinoceros beetles on its front tire. Three dung beetles roll balls past its tracks in the dirt, while acacia trees stand in the background.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, A glass Coca-Cola bottle lying sideways on brown leaves and soil. A line of black ants traverses its red label, two iridescent beetles explore the metal cap, and a pale moth rests on the glass neck.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, Black over-ear headphones on a wooden table. Three small beetles crawl along the ear cushions, while a spider hangs between the headband adjusters, its web gleaming in the light.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, A white athletic shoe on packed earth. Carpenter ants march through its eyelets, a beetle rests under the loosened tongue, and a cricket perches on the heel.

Negative Prompt

blurry, cropped, ugly

Prompt

In the style of an Alfred Edmund Brehm illustration, Three wooden pencils lying across a blank paper sheet. A praying mantis stands on one pencil tip, while two ladybugs explore graphite shavings scattered below.

Negative Prompt

blurry, cropped, ugly

The text encoder was not trained. You may reuse the base model text encoder for inference.

Training settings

• Training epochs: 0

• Training steps: 600

• Learning rate: 0.0004

  • Learning rate schedule: polynomial
  • Warmup steps: 200

• Max grad norm: 2.0

• Effective batch size: 3

  • Micro-batch size: 3
  • Gradient accumulation steps: 1
  • Number of GPUs: 1

• Gradient checkpointing: True

• Prediction type: flow-matching (extra parameters=['shift=3', 'flux_guidance_mode=constant', 'flux_guidance_value=1.0', 'flow_matching_loss=compatible'])

• Optimizer: adamw_bf16

• Trainable parameter precision: Pure BF16

• Caption dropout probability: 10.0%

• SageAttention: Enabled inference

LyCORIS Config:

{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 16,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 16
            },
            "FeedForward": {
                "factor": 8
            }
        }
    }
}

Datasets

ab-512

• Repeats: 12
• Total number of images: 74
• Total number of aspect buckets: 4
• Resolution: 0.262144 megapixels
• Cropped: False
• Crop style: None
• Crop aspect: None
• Used for regularisation data: No

ab-768

• Repeats: 8
• Total number of images: 74
• Total number of aspect buckets: 9
• Resolution: 0.589824 megapixels
• Cropped: False
• Crop style: None
• Crop aspect: None
• Used for regularisation data: No

ab-1024

• Repeats: 5
• Total number of images: 74
• Total number of aspect buckets: 11
• Resolution: 1.048576 megapixels
• Cropped: False
• Crop style: None
• Crop aspect: None
• Used for regularisation data: No

ab-1536

• Repeats: 2
• Total number of images: 73
• Total number of aspect buckets: 16
• Resolution: 2.359296 megapixels
• Cropped: False
• Crop style: None
• Crop aspect: None
• Used for regularisation data: No

ab-crops-512

• Repeats: 8
• Total number of images: 74
• Total number of aspect buckets: 1
• Resolution: 0.262144 megapixels
• Cropped: True
• Crop style: random
• Crop aspect: square
• Used for regularisation data: No

ab-1024-crop

• Repeats: 6
• Total number of images: 74
• Total number of aspect buckets: 1
• Resolution: 1.048576 megapixels
• Cropped: True
• Crop style: random
• Crop aspect: square
• Used for regularisation data: No

Inference

import torch
from diffusers import DiffusionPipeline
from lycoris import create_lycoris_from_weights


def download_adapter(repo_id: str):
    import os
    from huggingface_hub import hf_hub_download
    adapter_filename = "pytorch_lora_weights.safetensors"
    cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models'))
    cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_")
    path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path)
    path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename)
    os.makedirs(path_to_adapter, exist_ok=True)
    hf_hub_download(
        repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter
    )

    return path_to_adapter_file
    
model_id = 'black-forest-labs/FLUX.1-dev'
adapter_repo_id = 'davidrd123/AlbertBierstadt-Flux-LoKr'
adapter_filename = 'pytorch_lora_weights.safetensors'
adapter_file_path = download_adapter(repo_id=adapter_repo_id)
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
lora_scale = 1.0
wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer)
wrapper.merge_to()

prompt = "An astronaut is riding a horse through the jungles of Thailand."


## Optional: quantise the model to save on vram.
## Note: The model was quantised during training, and so it is recommended to do the same during inference time.
from optimum.quanto import quantize, freeze, qint8
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
    
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
    prompt=prompt,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
    width=1024,
    height=1280,
    guidance_scale=3.0,
).images[0]
image.save("output.png", format="PNG")

Exponential Moving Average (EMA)

SimpleTuner generates a safetensors variant of the EMA weights and a pt file.

The safetensors file is intended to be used for inference, and the pt file is for continuing finetuning.

The EMA model may provide a more well-rounded result, but typically will feel undertrained compared to the full model as it is a running decayed average of the model weights.