paulprt/george_barbier_qlora_flux

LoRA for FLUX.1-Dev - George Barbier Style

This model was built following this tutorial which explains in detail how to fine-tune FLUX.1-dev. I copied and adapted the repository page from the tutorial. Original can be found here

This repository contains a LoRA (Low-Rank Adaptation) fine-tuned on black-forest-labs/FLUX.1-dev to generate images in the artistic style of George Barbier. This work follows the blog post, "Fine-Tuning FLUX.1-dev on consumer hardware and in FP8".

Prompt

A man holding a cigare next to a window, george barbier style

Prompt

A woman next to a tree, george barbier style

Prompt

A woman dancing in blue dress, george barbier style

Model Description

This LoRA was trained on paulprt/george-barbier-style dataset.

Inference

There are two main ways to use this LoRA for inference: loading the adapter on the fly or merging it with the base model.

Option 1: Loading LoRA Adapters

This approach offers flexibility, allowing you to easily switch between different LoRA styles.

from diffusers import FluxPipeline
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, torch_dtype=torch.float16
)
pipeline.load_lora_weights("paulprt/george-barbier-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.enable_model_cpu_offload()

image = pipeline(
    "Two women dancing in nature, george barbier style",
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("george_barbier_loaded.png")

Option 2: Merging LoRA into Base Model

Merging the LoRA into the base model can lead to slightly faster inference and is useful when you want to use a single style consistently.

from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("paulprt/george-barbier-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

# You can save the fused transformer for later use
# pipeline.transformer.save_pretrained("fused_transformer")

pipeline.enable_model_cpu_offload()
image = pipeline(
    "Two women dancing in nature, george barbier style",
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("george_barbier_merged.png")

you can also requantize model:

from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel, BitsAndBytesConfig
import torch 

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("paulprt/george-barbier-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

pipeline.transformer.save_pretrained("fused_transformer")

ckpt_id = "black-forest-labs/FLUX.1-dev"
bnb_4bit_compute_dtype = torch.bfloat16

nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=bnb_4bit_compute_dtype,
)
transformer = FluxTransformer2DModel.from_pretrained(
    "fused_transformer",
    quantization_config=nf4_config,
    torch_dtype=bnb_4bit_compute_dtype,
)

pipeline = AutoPipelineForText2Image.from_pretrained(
    ckpt_id, transformer=transformer, torch_dtype=bnb_4bit_compute_dtype
)
pipeline.enable_model_cpu_offload()

image = pipeline(
    "Two women dancing in nature, george barbier style",
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("george_barbier_merged.png")

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers

License

Please adhere to the licensing terms as described here.