---
pipeline_tag: text-to-image
license: other
license_name: sai-nc-community
license_link: https://huggingface.co/stabilityai/sd-turbo/blob/main/LICENSE.TXT
base_model: stabilityai/sd-turbo
language:
  - en
tags:
  - stable-diffusion
  - sdxl
  - onnxruntime
  - onnx
  - text-to-image
---

# Stable Diffusion Turbo for ONNX Runtime CUDA

## Introduction

This repository hosts the optimized ONNX models of **SD Turbo** to accelerate inference with ONNX Runtime CUDA execution provider for Nvidia GPUs. It cannot run in other providers like CPU and DirectML.

The models are generated by [Olive](https://github.com/microsoft/Olive/tree/main/examples/stable_diffusion) with command like the following:
```
python stable_diffusion.py --provider cuda --model_id stabilityai/sd-turbo --optimize --use_fp16_fixed_vae
```

See the [usage instructions](#usage-example) for how to run the SDXL pipeline with the ONNX files hosted in this repository.

## Model Description

- **Developed by:** Stability AI
- **Model type:** Diffusion-based text-to-image generative model
- **License:** [STABILITY AI NON-COMMERCIAL RESEARCH COMMUNITY LICENSE](https://huggingface.co/stabilityai/sd-turbo/blob/main/LICENSE)
- **Model Description:** This is a conversion of the [SD-Turbo](https://huggingface.co/stabilityai/sd-turbo) model for [ONNX Runtime](https://github.com/microsoft/onnxruntime) inference with CUDA execution provider.

## Performance

#### Latency

Below is average latency of generating an image of size 512x512 using NVIDIA A100-SXM4-80GB GPU:

| Engine      | Batch Size | Steps | ONNX Runtime CUDA |
|-------------|------------|------ | ----------------- |
| Static      | 1          |   1   | 38.2 ms           |
| Static      | 4          |   1   | 120.2 ms          |
| Static      | 1          |   4   | 68.7 ms           |
| Static      | 4          |   4   | 192.6 ms          |


Static means the engine is built for the given batch size and image size combination, and CUDA graph is used to speed up.


## Usage Example

Following the [demo instructions](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/stable_diffusion/README.md#run-demo-with-docker). Example steps:

0. Install nvidia-docker using these [instructions](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).

1. Clone onnxruntime repository.
```shell
git clone https://github.com/microsoft/onnxruntime
cd onnxruntime
```

2. Download the ONNX files from this repo
```shell
git lfs install
git clone https://huggingface.co/tlwu/sd-turbo-onnxruntime
```

3. Launch the docker
```shell
docker run --rm -it --gpus all -v $PWD:/workspace nvcr.io/nvidia/pytorch:23.10-py3 /bin/bash
```

4. Build ONNX Runtime from source
```shell
export CUDACXX=/usr/local/cuda-12.2/bin/nvcc
git config --global --add safe.directory '*'
sh build.sh --config Release  --build_shared_lib --parallel --use_cuda --cuda_version 12.2 \
            --cuda_home /usr/local/cuda-12.2 --cudnn_home /usr/lib/x86_64-linux-gnu/ --build_wheel --skip_tests \
            --use_tensorrt --tensorrt_home /usr/src/tensorrt \
            --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF \
            --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 \
            --allow_running_as_root
python3 -m pip install build/Linux/Release/dist/onnxruntime_gpu-*-cp310-cp310-linux_x86_64.whl --force-reinstall
```

If the GPU is not A100, change CMAKE_CUDA_ARCHITECTURES=80 in the command line according to the GPU compute capacity (like 89 for RTX 4090, or 86 for RTX 3090). If your machine has less than 64GB memory, replace --parallel by --parallel 4 --nvcc_threads 1  to avoid out of memory.

5. Install libraries and requirements
```shell
python3 -m pip install --upgrade pip
cd /workspace/onnxruntime/python/tools/transformers/models/stable_diffusion
python3 -m pip install -r requirements-cuda12.txt
python3 -m pip install --upgrade polygraphy onnx-graphsurgeon --extra-index-url https://pypi.ngc.nvidia.com
```

6. Perform ONNX Runtime optimized inference
```shell
python3 demo_txt2img.py \
  "starry night over Golden Gate Bridge by van gogh" \
  --version sd-turbo   \
  --engine-dir /workspace/sd-turbo-onnxruntime
```