File size: 3,980 Bytes
1bfa61c f668844 1bfa61c f668844 1bfa61c 019ea83 1bfa61c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 |
## Generate Example:
- Model: [andite/anything-v4.0](https://hf.co/andite/anything-v4.0)
- Prompt: `best quality, extremely detailed, cowboy shot`
- Negative_prompt: `cowboy, monochrome, lowres, bad anatomy, worst quality, low quality`
- Seed: 19 (cherry-picked)
|Control Image1|Control Image2|Generated|
|---|---|---|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png">|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png">|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_pose_and_canny_result_19.png">|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png">|(none)|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_pose_only_result_19.png">|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png">|(none)|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_canny_only_result_19.png">|
- Pose & Canny Control Image Generated by [*Character bones that look like Openpose for blender _ Ver_4.7 Depth+Canny*](https://toyxyz.gumroad.com/l/ciojz)
## Code
Using deveploment version [`StableDiffusionMultiControlNetPipeline`](https://github.com/takuma104/diffusers/tree/multi_controlnet)
```py
from stable_diffusion_multi_controlnet import StableDiffusionMultiControlNetPipeline
from stable_diffusion_multi_controlnet import ControlNetProcessor
from diffusers import ControlNetModel
import torch
from diffusers.utils import load_image
pipe = StableDiffusionMultiControlNetPipeline.from_pretrained(
"andite/anything-v4.0", safety_checker=None, torch_dtype=torch.float16
).to("cuda")
pipe.scheduler = EulerDiscreteScheduler.from_config("andite/anything-v4.0", subfolder="scheduler")
pipe.enable_xformers_memory_efficient_attention()
controlnet_canny = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
).to("cuda")
controlnet_pose = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16
).to("cuda")
canny_image = load_image('https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png').convert('RGB')
pose_image = load_image('https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png').convert('RGB')
prompt = "best quality, extremely detailed, cowboy shot"
negative_prompt = "cowboy, monochrome, lowres, bad anatomy, worst quality, low quality"
seed = 19
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
processors=[
ControlNetProcessor(controlnet_pose, pose_image),
# ControlNetProcessor(controlnet_canny, canny_image),
],
generator=torch.Generator(device="cpu").manual_seed(seed),
num_inference_steps=30,
width=512,
height=512,
).images[0]
image.save(f"./mc_pose_only_result_{seed}.png")
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
processors=[
# ControlNetProcessor(controlnet_pose, pose_image),
ControlNetProcessor(controlnet_canny, canny_image),
],
generator=torch.Generator(device="cpu").manual_seed(seed),
num_inference_steps=30,
width=512,
height=512,
).images[0]
image.save(f"./mc_canny_only_result_{seed}.png")
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
processors=[
ControlNetProcessor(controlnet_pose, pose_image),
ControlNetProcessor(controlnet_canny, canny_image),
],
generator=torch.Generator(device="cpu").manual_seed(seed),
num_inference_steps=30,
width=512,
height=512,
).images[0]
image.save(f"./mc_pose_and_canny_result_{seed}.png")
```
|