sayakpaul HF staff commited on
Commit
44dbf89
1 Parent(s): cd9b022

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -16
README.md CHANGED
@@ -10,9 +10,10 @@ tags:
10
  inference: false
11
  ---
12
 
13
- # SDXL-controlnet: Canny
14
 
15
- These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with canny conditioning. You can find some example images in the following.
 
16
 
17
  prompt: aerial view, a futuristic research complex in a bright foggy jungle, hard lighting
18
  ![images_0)](./cann-small-hf-ofice.png)
@@ -82,14 +83,21 @@ images[0].save(f"hug_lab.png")
82
 
83
  To more details, check out the official documentation of [`StableDiffusionXLControlNetPipeline`](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl).
84
 
85
- 🚨 Please note that this checkpoint is experimental and should be deeply investigated before being deployed. We encourage the community to build on top
86
- of it and improve it. 🚨
87
 
88
  ### Training
89
 
90
  Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md).
91
  You can refer to [this script](https://github.com/patil-suraj/muse-experiments/blob/f71e7e79af24509ddb4e1b295a1d0ef8d8758dc9/ctrlnet/train_controlnet_webdataset.py) for full discolsure.
92
 
 
 
 
 
 
 
 
 
93
  #### Training data
94
  The model was trained on 3M images from LAION aesthetic 6 plus subset, with batch size of 256 for 50k steps with constant learning rate of 3e-5.
95
 
@@ -97,15 +105,4 @@ The model was trained on 3M images from LAION aesthetic 6 plus subset, with batc
97
  One 8xA100 machine
98
 
99
  #### Mixed precision
100
- FP16
101
-
102
- #### Additional notes
103
-
104
- * This checkpoint does not perform distillation. We just use a smaller ControlNet initialized from the SDXL UNet. We
105
- encourage the community to try and conduct distillation too, where the smaller ControlNet model would be initialized from
106
- a bigger ControlNet model. This resource might be of help in [this regard](https://huggingface.co/blog/sd_distillation).
107
- * It does not have any attention blocks.
108
- * It is better suited for simple conditioning images. For conditionings involving more complex structures, you
109
- should use the bigger checkpoints.
110
- * We recommend playing around with the `controlnet_conditioning_scale` and `guidance_scale` arguments for potentially better
111
- image generation quality.
 
10
  inference: false
11
  ---
12
 
13
+ # Small SDXL-controlnet: Canny
14
 
15
+ These are small controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with canny conditioning. This checkpoint is 7x smaller than the original XL controlnet checkpoint.
16
+ You can find some example images in the following.
17
 
18
  prompt: aerial view, a futuristic research complex in a bright foggy jungle, hard lighting
19
  ![images_0)](./cann-small-hf-ofice.png)
 
83
 
84
  To more details, check out the official documentation of [`StableDiffusionXLControlNetPipeline`](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl).
85
 
86
+ 🚨 Please note that this checkpoint is experimental and there's a lot of room for improvement. We encourage the community to build on top of it, improve it, and provide us with feedback. 🚨
 
87
 
88
  ### Training
89
 
90
  Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md).
91
  You can refer to [this script](https://github.com/patil-suraj/muse-experiments/blob/f71e7e79af24509ddb4e1b295a1d0ef8d8758dc9/ctrlnet/train_controlnet_webdataset.py) for full discolsure.
92
 
93
+ * This checkpoint does not perform distillation. We just use a smaller ControlNet initialized from the SDXL UNet. We
94
+ encourage the community to try and conduct distillation too. This resource might be of help in [this regard](https://huggingface.co/blog/sd_distillation).
95
+ * To learn more about how the ControlNet was initialized, refer to [this code block](https://github.com/patil-suraj/muse-experiments/blob/f71e7e79af24509ddb4e1b295a1d0ef8d8758dc9/ctrlnet/train_controlnet_webdataset.py#L1020C1-L1042C36).
96
+ * It does not have any attention blocks.
97
+ * The model works pretty good on most conditioning images. But for more complex conditionings, the bigger checkpoints might be better. We are still working on improving the quality of this checkpoint and looking for feedback from the community.
98
+ * We recommend playing around with the `controlnet_conditioning_scale` and `guidance_scale` arguments for potentially better
99
+ image generation quality.
100
+
101
  #### Training data
102
  The model was trained on 3M images from LAION aesthetic 6 plus subset, with batch size of 256 for 50k steps with constant learning rate of 3e-5.
103
 
 
105
  One 8xA100 machine
106
 
107
  #### Mixed precision
108
+ FP16