|
--- |
|
license: mit |
|
tags: |
|
- stable-diffusion |
|
- stable-diffusion-diffusers |
|
inference: false |
|
--- |
|
# SDXL - VAE |
|
|
|
#### How to use with 🧨 diffusers |
|
You can integrate this fine-tuned VAE decoder to your existing `diffusers` workflows, by including a `vae` argument to the `StableDiffusionPipeline` |
|
```py |
|
from diffusers.models import AutoencoderKL |
|
from diffusers import StableDiffusionPipeline |
|
|
|
model = "stabilityai/your-stable-diffusion-model" |
|
vae = AutoencoderKL.from_pretrained("stabilityai/sdxl-vae") |
|
pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae) |
|
``` |
|
|
|
## Model |
|
[SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9) is a [latent diffusion model](https://arxiv.org/abs/2112.10752), where the diffusion operates in a pretrained, |
|
learned (and fixed) latent space of an autoencoder. |
|
While the bulk of the semantic composition is done by the latent diffusion model, |
|
we can improve _local_, high-frequency details in generated images by improving the quality of the autoencoder. |
|
To this end, we train the same autoencoder architecture used for the original [Stable Diffusion](https://github.com/CompVis/stable-diffusion) at a larger batch-size (256 vs 9) |
|
and additionally track the weights with an exponential moving average (EMA). |
|
The resulting autoencoder outperforms the original model in all evaluated reconstruction metrics, see the table below. |
|
|
|
|
|
## Evaluation |
|
_SDXL-VAE vs original kl-f8 VAE vs f8-ft-MSE_ |
|
### COCO 2017 (256x256, val, 5000 images) |
|
| Model | rFID | PSNR | SSIM | PSIM | Link | Comments |
|
|----------|------|--------------|---------------|---------------|------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------| |
|
| | | | | | | | |
|
| SDXL-VAE | 4.42 | 24.7 +/- 3.9 | 0.73 +/- 0.13 | 0.88 +/- 0.27 | https://huggingface.co/stabilityai/sdxl-vae/blob/main/sdxl_vae.safetensors | as used in SDXL | |
|
| original | 4.99 | 23.4 +/- 3.8 | 0.69 +/- 0.14 | 1.01 +/- 0.28 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD | |
|
| ft-MSE | 4.70 | 24.5 +/- 3.7 | 0.71 +/- 0.13 | 0.92 +/- 0.27 | https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs | |
|
|
