promeai/FLUX.1-controlnet-lineart-promeai

promeai/FLUX.1-controlnet-lineart-promeai holds controlnet weights trained on black-forest-labs/FLUX.1-dev with lineart condition.

Here are some example images.

prompt: cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere

input-control	result image

How to use

With diffusers

import torch
from diffusers.utils import load_image
from diffusers.pipelines.flux.pipeline_flux_controlnet import FluxControlNetPipeline
from diffusers.models.controlnet_flux import FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model = 'promeai/FLUX.1-controlnet-lineart-promeai'
controlnet = FluxControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

control_image = load_image("./images/example-control.jpg")
prompt = "cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere"
image = pipe(
    prompt, 
    control_image=control_image,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=28, 
    guidance_scale=3.5,
).images[0]
image.save("./image.jpg")

With comfyui

An example comfyui workflow is also provided.

Limitations and bias

• This model is not intended or able to provide factual information.
• As a statistical model this checkpoint might amplify existing societal biases.
• The model may fail to generate output that matches the prompts.
• Prompt following is heavily influenced by the prompting-style.

Training details

This controlnet is trained on one A100-80G GPU, with carefully selected proprietary real-world images dataset, with imagesize 512 + batchsize 3 (earlier period), and imagesize 1024 + batchsize 1 (after 512 training). With above configs, the GPU memory was about 70G and takes around 3 days to get this 14000steps-checkpoint.