valhalla commited on
Commit
57f7e66
1 Parent(s): 3f0be04

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +111 -0
README.md ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+ license: openrail++
4
+ base_model: stabilityai/stable-diffusion-xl-base-1.0
5
+ tags:
6
+ - stable-diffusion-xl
7
+ - stable-diffusion-xl-diffusers
8
+ - text-to-image
9
+ - diffusers
10
+ - controlnet
11
+ inference: false
12
+ ---
13
+
14
+ # SDXL-controlnet: Depth
15
+
16
+ These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with depth conditioning. You can find some example images in the following.
17
+
18
+ prompt: spiderman lecture, photorealistic
19
+ ![images_0)](./spiderman.png)
20
+
21
+ ## Usage
22
+
23
+ Make sure to first install the libraries:
24
+
25
+ ```bash
26
+ pip install accelerate transformers safetensors diffusers
27
+ ```
28
+
29
+ And then we're ready to go:
30
+
31
+ ```python
32
+ import torch
33
+ import numpy as np
34
+ from PIL import Image
35
+
36
+ from transformers import DPTFeatureExtractor, DPTForDepthEstimation
37
+ from diffusers import ControlNetModel, StableDiffusionXLControlNetPipeline, AutoencoderKL
38
+ from diffusers.utils import load_image
39
+
40
+
41
+ depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")
42
+ feature_extractor = DPTFeatureExtractor.from_pretrained("Intel/dpt-hybrid-midas")
43
+ controlnet = ControlNetModel.from_pretrained(
44
+ "diffusers/controlnet-depth-sdxl-1.0",
45
+ variant="fp16",
46
+ use_safetensors=True,
47
+ torch_dtype=torch.float16,
48
+ ).to("cuda")
49
+ vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16).to("cuda")
50
+ pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
51
+ "stabilityai/stable-diffusion-xl-base-1.0",
52
+ controlnet=controlnet,
53
+ vae=vae,
54
+ variant="fp16",
55
+ use_safetensors=True,
56
+ torch_dtype=torch.float16,
57
+ ).to("cuda")
58
+ pipe.enable_model_cpu_offload()
59
+
60
+ def get_depth_map(image):
61
+ image = feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")
62
+ with torch.no_grad(), torch.autocast("cuda"):
63
+ depth_map = depth_estimator(image).predicted_depth
64
+
65
+ depth_map = torch.nn.functional.interpolate(
66
+ depth_map.unsqueeze(1),
67
+ size=(1024, 1024),
68
+ mode="bicubic",
69
+ align_corners=False,
70
+ )
71
+ depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
72
+ depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
73
+ depth_map = (depth_map - depth_min) / (depth_max - depth_min)
74
+ image = torch.cat([depth_map] * 3, dim=1)
75
+
76
+ image = image.permute(0, 2, 3, 1).cpu().numpy()[0]
77
+ image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))
78
+ return image
79
+
80
+
81
+ prompt = "stormtrooper lecture, photorealistic"
82
+ image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-depth/resolve/main/images/stormtrooper.png")
83
+ controlnet_conditioning_scale = 0.5 # recommended for good generalization
84
+
85
+ depth_image = get_depth_map(image)
86
+
87
+ images = pipe(
88
+ prompt, image=depth_image, num_inference_steps=30, controlnet_conditioning_scale=controlnet_conditioning_scale,
89
+ ).images
90
+ images[0]
91
+
92
+ images[0].save(f"stormtrooper.png")
93
+ ```
94
+
95
+ To more details, check out the official documentation of [`StableDiffusionXLControlNetPipeline`](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl).
96
+
97
+ ### Training
98
+
99
+ Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md).
100
+
101
+ #### Training data and Compute
102
+ The model is trained on 3M image-text pairs from LAION-Aesthetics V2. The model is trained for 700 GPU hours on 80GB A100 GPUs.
103
+
104
+ #### Batch size
105
+ Data parallel with a single gpu batch size of 8 for a total batch size of 256.
106
+
107
+ #### Hyper Parameters
108
+ Constant learning rate of 1e-5.
109
+
110
+ #### Mixed precision
111
+ fp16