Hugging Face's logo

stabilityai
/
sd-vae-ft-mse

Diffusers
Safetensors
stable-diffusion
stable-diffusion-diffusers
Model card Files Community
12
multimodalart HF staff commited on
Commit
ea751d9
•
1 Parent(s): a3d9d34

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +81 -0
README.md CHANGED
@@ -1,3 +1,84 @@
1
  ---
2
  license: mit
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ tags:
4
+ - stable-diffusion
5
+ - stable-diffusion-diffusers
6
+ - text-to-image
7
+ inference: false
8
  ---
9
+ # Improved Autoencoders
10
+
11
+ ## Utilizing
12
+ These weights are intended to be used with the [🧨 diffusers library](https://github.com/huggingface/diffusers). If you are looking for the model to use with the original [CompVis Stable Diffusion codebase](https://github.com/CompVis/stable-diffusion), [come here](https://huggingface.co/CompVis/stabilityai/sd-vae-ft-ema-original).
13
+
14
+ #### How to use with 🧨 diffusers
15
+ You can integrate this fine-tuned VAE decoder to your existing `diffusers` workflows, by including a `vae` argument to the `StableDiffusionPipeline`
16
+ ```py
17
+ from diffusers.models import AutoencoderKL
18
+ from diffusers import StableDiffusionPipeline
19
+
20
+ model = "CompVis/stable-diffusion-v1-4"
21
+ vae = AutoencoderKL("stabilityai/sd-vae-ft-mse")
22
+ pipe = StableDiffusionPipeline(model, vae=vae)
23
+ ```
24
+
25
+ ## Decoder Finetuning
26
+ We publish two kl-f8 autoencoder versions, finetuned from the original [kl-f8 autoencoder](https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models).
27
+ The first, _ft-EMA_, was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights.
28
+ The second, _ft-MSE_, was resumed from _ft-EMA_ and uses EMA weights and was trained for another 280k steps using a re-weighted loss, with more emphasis
29
+ on MSE reconstruction (producing somewhat ``smoother'' outputs).
30
+ To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
31
+
32
+ _Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE_
33
+
34
+ ## Evaluation
35
+ ### COCO 2017 (256x256, val, 5000 images)
36
+ | Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments
37
+ |----------|---------|------|--------------|---------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
38
+ | | | | | | | | |
39
+ | original | 246803 | 4.99 | 23.4 +/- 3.8 | 0.69 +/- 0.14 | 1.01 +/- 0.28 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
40
+ | ft-EMA | 560001 | 4.42 | 23.8 +/- 3.9 | 0.69 +/- 0.13 | 0.96 +/- 0.27 | https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt | slightly better overall, with EMA |
41
+ | ft-MSE | 840001 | 4.70 | 24.5 +/- 3.7 | 0.71 +/- 0.13 | 0.92 +/- 0.27 | https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
42
+
43
+
44
+ ### LAION-Aesthetics 5+ (256x256, subset, 10000 images)
45
+ | Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments
46
+ |----------|-----------|------|--------------|---------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
47
+ | | | | | | | | |
48
+ | original | 246803 | 2.61 | 26.0 +/- 4.4 | 0.81 +/- 0.12 | 0.75 +/- 0.36 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
49
+ | ft-EMA | 560001 | 1.77 | 26.7 +/- 4.8 | 0.82 +/- 0.12 | 0.67 +/- 0.34 | https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt | slightly better overall, with EMA |
50
+ | ft-MSE | 840001 | 1.88 | 27.3 +/- 4.7 | 0.83 +/- 0.11 | 0.65 +/- 0.34 | https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
51
+
52
+
53
+ ### Visual
54
+ _Visualization of reconstructions on 256x256 images from the COCO2017 validation dataset._
55
+
56
+ <p align="center">
57
+ <br>
58
+ <b>
59
+ 256x256: ft-EMA (left), ft-MSE (middle), original (right)</b>
60
+ </p>
61
+
62
+ <p align="center">
63
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00025_merged.png />
64
+ </p>
65
+
66
+ <p align="center">
67
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00011_merged.png />
68
+ </p>
69
+
70
+ <p align="center">
71
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00037_merged.png />
72
+ </p>
73
+
74
+ <p align="center">
75
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00043_merged.png />
76
+ </p>
77
+
78
+ <p align="center">
79
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00053_merged.png />
80
+ </p>
81
+
82
+ <p align="center">
83
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/merged/00029_merged.png />
84
+ </p>