---
language:
- en
license: creativeml-openrail-m
library_name: diffusers
tags:
- art
- diffusion
- Interior
---

# KuJiaLe Layout ControlNet

Given the structural elements of the room, such as the walls, floors, and ceilings. Our model auto-completes the furnishing of the room.
The model is trained on [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) for interior designs.

### Layout ControlNet Example
Keep the room layout consistent, re-furnish the room.
<table style="table-layout: fixed;">
  <tr>
    <td style="text-align: center; vertical-align: middle; width: 50%">Input</td>
    <td style="text-align: center; vertical-align: middle; width: 50%">Output</td>
  </td>
  <tr>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_input.jpg"  alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/layout_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
  </tr> 
  <tr>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_input.jpg"  alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_and_dining_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
  </tr> 
  <tr>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_input.png"  alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/living_room_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
  </tr> 
  <tr>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_input.jpg"  alt="Input" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
    <td style="text-align: center; vertical-align: middle; width: 50%"> <img src="examples/kitchen_output.jpg" alt="Output" width="100%" style="display: block; margin-left: auto; margin-right: auto;"></td>
  </tr> 
  </tr>
</table>

## News🔥🔥🔥

* June.06, 2024. Our checkpoint Layout-ControlNet are publicly available on [HuggingFace Repo](https://huggingface.co/kujiale-ai/controlnet-layout).
* June.06, 2024. Our Layout-ControlNet demo are publicly available on [HuggingFace Space](https://huggingface.co/spaces/ysmao/Layout-Control).

## Try our HuggingFace demo: 
[HuggingFace Space Demo](https://huggingface.co/spaces/ysmao/Layout-Control)

## Checkpoints
* `control_v1_sd15_layout_fp16`: Layout ControlNet checkpoint, for SD15 models.

## Using in 🧨 diffusers
### Layout ControlNet
```python
import torch
from diffusers.utils import load_image
import numpy as np
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, UniPCMultistepScheduler

controlnet_checkpoint = "kujiale-ai/controlnet-layout"
# Load original image
image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_input.jpg")
depth_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_depth.jpg").convert("L")
normal_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_normal.jpg")
segm_image = load_image("https://huggingface.co/kujiale-ai/controlnet-layout/resolve/main/examples/layout_segm.jpg")
W, H = image.size
depth_image = depth_image.resize((W, H))
normal_image = normal_image.resize((W, H))
segm_image = segm_image.resize((W, H))
# Prepare Layout Control Image
depth_image = np.array(depth_image, dtype=np.float32) / 255.0
depth_image = torch.from_numpy(depth_image[:, :, None])[None].permute(0, 3, 1, 2)
normal_image = np.array(normal_image, dtype=np.float32)
normal_image = normal_image / 127.5 - 1.0
normal_image = torch.from_numpy(normal_image)[None].permute(0, 3, 1, 2)
segm_image = np.array(segm_image, dtype=np.float32) / 255.0
segm_image = torch.from_numpy(segm_image)[None].permute(0, 3, 1, 2)
control_image = torch.cat([depth_image, normal_image, segm_image], dim=1)
# Initialize pipeline
controlnet = ControlNetModel.from_pretrained(controlnet_checkpoint, subfolder="control_v1_sd15_layout_fp16", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
image = pipe("A modern bedroom,best quality", num_inference_steps=30, image=control_image, guidance_scale=7).images[0]
image.save('layout_output.jpg')
```