kujiale-ai/controlnet-layout

KuJiaLe Layout ControlNet

Given the structural elements of the room, such as the walls, floors, and ceilings. Our model auto-completes the furnishing of the room. The model is trained on runwayml/stable-diffusion-v1-5 for interior designs.

Layout ControlNet Example

Keep the room layout consistent, re-furnish the room.

Input	Output

News🔥🔥🔥

• June.06, 2024. Our checkpoint Layout-ControlNet are publicly available on HuggingFace Repo.
• June.06, 2024. Our Layout-ControlNet demo are publicly available on HuggingFace Space.

Try our HuggingFace demo:

HuggingFace Space Demo

Checkpoints

• control_v1_sd15_layout_fp16: Layout ControlNet checkpoint, for SD15 models.

Using in 🧨 diffusers

Layout ControlNet

import torch
from diffusers.utils import load_image
import numpy as np
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, UniPCMultistepScheduler

controlnet_checkpoint = "kujiale-ai/controlnet-layout"
# Load original image
image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_input.jpg")
depth_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_depth.jpg").convert("L")
normal_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_normal.jpg")
segm_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_segm.jpg")
W, H = image.size
depth_image = depth_image.resize((W, H))
normal_image = normal_image.resize((W, H))
segm_image = segm_image.resize((W, H))
# Prepare Layout Control Image
depth_image = np.array(depth_image, dtype=np.float32) / 255.0
depth_image = torch.from_numpy(depth_image[:, :, None])[None].permute(0, 3, 1, 2)
normal_image = np.array(normal_image, dtype=np.float32)
normal_image = normal_image / 127.5 - 1.0
normal_image = torch.from_numpy(normal_image)[None].permute(0, 3, 1, 2)
segm_image = np.array(segm_image, dtype=np.float32) / 255.0
segm_image = torch.from_numpy(segm_image)[None].permute(0, 3, 1, 2)
control_image = torch.cat([depth_image, normal_image, segm_image], dim=1)
# Initialize pipeline
controlnet = ControlNetModel.from_pretrained(controlnet_checkpoint, subfolder="control_v1_sd15_layout_fp16", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
image = pipe("A modern bedroom,best quality", num_inference_steps=30, image=control_image, guidance_scale=7).images[0]
image.save('layout_output.jpg')