---
license: other
license_name: stabilityai-ai-community
license_link: LICENSE.md
tags:
- stable-diffusion
- controlnet
inference: true
extra_gated_prompt: >-
  By clicking "Agree", you agree to the [License
  Agreement](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
  and acknowledge Stability AI's [Privacy
  Policy](https://stability.ai/privacy-policy).
extra_gated_fields:
  Name: text
  Email: text
  Country: country
  Organization or Affiliation: text
  Receive email updates and promotions on Stability AI products, services, and research?:
    type: select
    options:
    - 'Yes'
    - 'No'
  What do you intend to use the model for?: 
    type: select
    options:
    - Research
    - Personal use
    - Creative Professional
    - Startup
    - Enterprise
  I agree to the License Agreement and acknowledge Stability AI's Privacy Policy: checkbox

language:
- en
pipeline_tag: text-to-image
---

# Stable Diffusion 3.5 Large Controlnet - Depth
![ControlNet Demo Image](header.png)

## Model

This repository provides the Depth ControlNet for [Stable Diffusion 3.5 Large.](https://stability.ai/news/introducing-stable-diffusion-3-5).

Please note: This model is released under the [Stability Community License](https://stability.ai/community-license-agreement). Visit [Stability AI](https://stability.ai/license) to learn or [contact us](https://stability.ai/enterprise) for commercial licensing details.


### License

- **Community License:**  Free for research, non-commercial, and commercial use for organizations or individuals with less than $1M in total annual revenue. More details can be found in the [Community License Agreement](https://stability.ai/community-license-agreement). Read more at https://stability.ai/license.
- **For individuals and organizations with annual revenue above $1M**: please [contact us](https://stability.ai/enterprise) to get an Enterprise License.

## Usage

For now, we recommend using the [standalone SD3.5 repo](https://github.com/Stability-AI/sd3.5) to use the ControlNets.

A full technical report on Stable Diffusion 3.5, with details on the ControlNet training, will be released soon as well.

### Using Controlnets in SD3.5 Standalone Repo
Install the repo:
```
git clone git@github.com:Stability-AI/sd3.5.git
pip install -r requirements.txt
```

Then, download the models and sample image like so:

```
input/sample_cond.png
models/clip_g.safetensors
models/clip_l.safetensors
models/t5xxl.safetensors
models/sd3.5_large.safetensors
models/canny_8b.safetensors
```

and then you can run
```
python sd3_infer.py --controlnet_ckpt models/depth_8b.safetensors --controlnet_cond_image input/sample_cond.png --prompt "A girl sitting in a cafe, cozy interior, HDR photograph"
```

Which should give you an image like below:

![A girl sitting in a cafe](sample_result.png)


### Preprocessing

An input image can be preprocessed for control use following the code snippet below. SD3.5 does not implement this behavior, so we recommend doing so in an external script beforehand.

```python
# install depthfm from https://github.com/CompVis/depth-fm
import torchvision.transforms as transforms
from depthfm.dfm import DepthFM
depthfm_model = DepthFM(ckpt_path=checkpoint_path)
depthfm_model.eval()

# assuming img is a PIL image
img = F.to_tensor(img)
c, h, w = img.shape
img = F.interpolate(img, (512, 512), mode='bilinear', align_corners=False)
with torch.no_grad():
  img = self.depthfm_model(img, num_steps=2, ensemble_size=4)
img = F.interpolate(img, (h, w), mode='bilinear', align_corners=False)
```

### Tips
- Euler sampler and a slightly higher step count (50-60) gives best results, especially with Canny.
- Pass `--text_encoder_device <device_name>` to load the text encoders directly to VRAM, which can speed up the full inference loop at the cost of extra VRAM usage.

## Uses

### Intended Uses

Intended uses include the following:
* Generation of artworks and use in design and other artistic processes.
* Applications in educational or creative tools.
* Research on generative models, including understanding the limitations of generative models.

All uses of the model must be in accordance with our [Acceptable Use Policy](https://stability.ai/use-policy).

### Training Data and Strategy
    
These models were trained on a wide variety of data, including synthetic data and filtered publicly available data.
The data used is a subset of the Stable Diffusion 3.5 post-training dataset, and satisfies the same legal and safety requirements.

### Out-of-Scope Uses

The model was not trained to be factual or true representations of people or events.  As such, using the model to generate such content is out-of-scope of the abilities of this model.

## Safety

As part of our safety-by-design and responsible AI deployment approach, we take deliberate measures to ensure Integrity starts at the early stages of development. We implement safety measures throughout the development of our models. We have implemented safety mitigations that are intended to reduce the risk of certain harms, however we recommend that developers conduct their own testing and apply additional mitigations based on their specific use cases.  
For more about our approach to Safety, please visit our [Safety page](https://stability.ai/safety).

### Integrity Evaluation

Our integrity evaluation methods include structured evaluations and red-teaming testing for certain harms.  Testing was conducted primarily in English and may not cover all possible harms.  

### Risks identified and mitigations:

* Harmful content:  We have used filtered data sets when training our models and implemented safeguards that attempt to strike the right balance between usefulness and preventing harm. However, this does not guarantee that all possible harmful content has been removed. All developers and deployers should exercise caution and implement content safety guardrails based on their specific product policies and application use cases.
* Misuse: Technical limitations and developer and end-user education can help mitigate against malicious applications of models. All users are required to adhere to our [Acceptable Use Policy](https://stability.ai/use-policy), including when applying fine-tuning and prompt engineering mechanisms. Please reference the Stability AI Acceptable Use Policy for information on violative uses of our products.
* Privacy violations: Developers and deployers are encouraged to adhere to privacy regulations with techniques that respect data privacy.

### Acknowledgements


- Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, authors of the original [ControlNet paper](https://arxiv.org/abs/2302.05543).
- Lvmin Zhang, who also developed the [Tile ControlNet](https://huggingface.co/lllyasviel/control_v11f1e_sd15_tile), which inspired the Blur ControlNet.
- [Diffusers](https://github.com/huggingface/diffusers) library authors, whose code was referenced during development.
- [InstantX](https://github.com/instantX-research) team, whose Flux and SD3 ControlNets were also referenced during training.
- All early testers and raters of the models, and the Stability AI team.

### Contact

Please report any issues with the model or contact us:

* Safety issues:  safety@stability.ai
* Security issues:  security@stability.ai
* Privacy issues:  privacy@stability.ai
* License and general: https://stability.ai/license
* Enterprise license: https://stability.ai/enterprise