Hugging Face's logo

promeai
/
FLUX.1-controlnet-lineart-promeai

Text-to-Image
Diffusers
Safetensors
flux
flux-diffusers
controlnet
diffusers-training
Model card Files Community
2
FLUX.1-controlnet-lineart-promeai / README.md
promeai's picture
promeai
update readme
0a54870 verified
preview code
|
raw
history blame
3.09 kB
metadata
base_model: black-forest-labs/FLUX.1-dev
library_name: diffusers
tags:
  - flux
  - flux-diffusers
  - text-to-image
  - diffusers
  - controlnet
  - diffusers-training
  - flux
  - flux-diffusers
  - text-to-image
  - diffusers
  - controlnet
  - diffusers-training
inference: true

promeai/FLUX.1-controlnet-lineart-promeai

promeai/FLUX.1-controlnet-lineart-promeai holds controlnet weights trained on black-forest-labs/FLUX.1-dev with lineart condition.

Here are some example images.

prompt: cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere

input-control) output)

Intended uses & limitations

How to use

with diffusers 

# TODO: add an example code snippet for running this diffusion pipeline
import torch
from diffusers.utils import load_image
from diffusers.pipelines.flux.pipeline_flux_controlnet import FluxControlNetPipeline
from diffusers.models.controlnet_flux import FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model = 'promeai/FLUX.1-controlnet-lineart-promeai'
controlnet = FluxControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

control_image = load_image("./images/example-control.jpg")
prompt = "cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere"
image = pipe(
    prompt, 
    control_image=control_image,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=28, 
    guidance_scale=3.5,
).images[0]
image.save("./image.jpg")

with comfyui

An example comfyui workflowis also provided.

Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

Training details

This controlnet is trained on one A100-80G GPU, with fine grained realword images dataset, with imagesize 512 + batchsize 3 (earlier period), and imagesize 1024 + batchsize 1 (after 512 training). With above configs, the GPU memory was about 70G and takes around 3 days to get this 14000steps-checkpoint. Training progress is going on, more ckpts will be released.