|
--- |
|
base_model: black-forest-labs/FLUX.1-dev |
|
library_name: diffusers |
|
tags: |
|
- flux |
|
- flux-diffusers |
|
- text-to-image |
|
- diffusers |
|
- controlnet |
|
- diffusers-training |
|
- flux |
|
- flux-diffusers |
|
- text-to-image |
|
- diffusers |
|
- controlnet |
|
- diffusers-training |
|
inference: true |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the training script had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# promeai/FLUX.1-controlnet-lineart-promeai |
|
|
|
`promeai/FLUX.1-controlnet-lineart-promeai` holds controlnet weights trained on black-forest-labs/FLUX.1-dev with lineart condition. |
|
|
|
|
|
Here are some example images. |
|
|
|
``` |
|
prompt: cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere |
|
``` |
|
 |
|
 |
|
|
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
## How to use |
|
|
|
### with diffusers |
|
|
|
```python |
|
# TODO: add an example code snippet for running this diffusion pipeline |
|
import torch |
|
from diffusers.utils import load_image |
|
from diffusers.pipelines.flux.pipeline_flux_controlnet import FluxControlNetPipeline |
|
from diffusers.models.controlnet_flux import FluxControlNetModel |
|
|
|
base_model = 'black-forest-labs/FLUX.1-dev' |
|
controlnet_model = 'promeai/FLUX.1-controlnet-lineart-promeai' |
|
controlnet = FluxControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16) |
|
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16) |
|
pipe.to("cuda") |
|
|
|
control_image = load_image("./images/example-control.jpg") |
|
prompt = "cute anime girl with massive fluffy fennec ears and a big fluffy tail blonde messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron mouth open holding a fancy black forest cake with candles on top in the kitchen of an old dark Victorian mansion lit by candlelight with a bright window to the foggy forest and very expensive stuff everywhere" |
|
image = pipe( |
|
prompt, |
|
control_image=control_image, |
|
controlnet_conditioning_scale=0.6, |
|
num_inference_steps=28, |
|
guidance_scale=3.5, |
|
).images[0] |
|
image.save("./image.jpg") |
|
``` |
|
|
|
### with comfyui |
|
An [example comfyui workflow](./example_workflow.json)is also provided. |
|
|
|
|
|
## Limitations and bias |
|
|
|
[TODO: provide examples of latent issues and potential remediations] |
|
|
|
## Training details |
|
|
|
This controlnet is trained on one A100-80G GPU, with fine grained realword images dataset, with imagesize 512 + batchsize 3 (earlier period), and imagesize 1024 + batchsize 1 (after 512 training). With above configs, the GPU memory was about 70G and takes around 3 days to get this 14000steps-checkpoint. Training progress is going on, more ckpts will be released. |
