Hugging Face's logo

stabilityai
/
sd-vae-ft-mse-original

Text-to-Image
stable-diffusion
stable-diffusion-diffusers
Model card Files Community
12
multimodalart HF staff commited on
Commit
83794be
•
1 Parent(s): 6a7c56f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +78 -0
README.md CHANGED
@@ -1,3 +1,81 @@
1
  ---
2
  license: creativeml-openrail-m
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: creativeml-openrail-m
3
+ tags:
4
+ - stable-diffusion
5
+ - stable-diffusion-diffusers
6
+ - text-to-image
7
+ inference: false
8
  ---
9
+ # Improved Autoencoders
10
+ ## Decoder Finetuning
11
+ We publish two kl-f8 autoencoder versions, finetuned from the original [kl-f8 autoencoder](https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models).
12
+ The first, _ft-EMA_, was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights.
13
+ The second, _ft-MSE_, was resumed from _ft-EMA_ and uses EMA weights and was trained for another 280k steps using a re-weighted loss, with more emphasis
14
+ on MSE reconstruction (producing somewhat ``smoother'' outputs).
15
+ To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
16
+
17
+ _Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE_
18
+
19
+ ## Evaluation
20
+ ### COCO 2017 (256x256, val, 5000 images)
21
+ | Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments
22
+ |----------|---------|------|--------------|---------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
23
+ | | | | | | | | |
24
+ | original | 246803 | 4.99 | 23.4 +/- 3.8 | 0.69 +/- 0.14 | 1.01 +/- 0.28 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
25
+ | ft-EMA | 560001 | 4.42 | 23.8 +/- 3.9 | 0.69 +/- 0.13 | 0.96 +/- 0.27 | https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/blob/main/ckpt_00560000-kat-ema-pruned.ckpt | slightly better overall, with EMA |
26
+ | ft-MSE | 840001 | 4.70 | 24.5 +/- 3.7 | 0.71 +/- 0.13 | 0.92 +/- 0.27 | https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/blob/main/ckpt_00840000-kat-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
27
+
28
+
29
+ ### LAION-Aesthetics 5+ (256x256, subset, 10000 images)
30
+ | Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments
31
+ |----------|-----------|------|--------------|---------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
32
+ | | | | | | | | |
33
+ | original | 246803 | 2.61 | 26.0 +/- 4.4 | 0.81 +/- 0.12 | 0.75 +/- 0.36 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
34
+ | ft-EMA | 560001 | 1.77 | 26.7 +/- 4.8 | 0.82 +/- 0.12 | 0.67 +/- 0.34 | https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/blob/main/ckpt_00560000-kat-ema-pruned.ckpt | slightly better overall, with EMA |
35
+ | ft-MSE | 840001 | 1.88 | 27.3 +/- 4.7 | 0.83 +/- 0.11 | 0.65 +/- 0.34 | https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/blob/main/ckpt_00840000-kat-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
36
+
37
+
38
+ ### Visual
39
+ _Visualization of reconstructions on 256x256 images from the COCO2017 validation dataset._
40
+
41
+ <p align="center">
42
+ <br>
43
+ <b>
44
+ 256x256: ft-MSE (top), ft-EMA (middle), original (right)</b>
45
+ </p>
46
+
47
+ <p align="center">
48
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00025.png" />
49
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00025.png" />
50
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00025.png" />
51
+ </p>
52
+
53
+ <p align="center">
54
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00011.png" />
55
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00011.png" />
56
+ <img src="https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00011.png" />
57
+ </p>
58
+
59
+ <p align="center">
60
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00037.png />
61
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00037.png />
62
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00037.png />
63
+ </p>
64
+
65
+ <p align="center">
66
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00043.png />
67
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00043.png />
68
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00043.png />
69
+ </p>
70
+
71
+ <p align="center">
72
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00053.png />
73
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00053.png />
74
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00053.png />
75
+ </p>
76
+
77
+ <p align="center">
78
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-mse/00029.png />
79
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/ft-ema/00029.png />
80
+ <img src=https://huggingface.co/stabilityai/stable-diffusion-decoder-finetune/resolve/main/eval/ae-decoder-tuning-reconstructions/original/00029.png />
81
+ </p>