Create README.md

README.md

@@ -0,0 +1,140 @@
+---
+license: creativeml-openrail-m
+tags:
+- stable-diffusion
+- stable-diffusion-diffusers
+- text-to-image
+datasets:
+- ChristophSchuhmann/improved_aesthetics_6.25plus
+library_name: diffusers
+pipeline_tag: text-to-image
+extra_gated_prompt: >-
+  This model is open access and available to all, with a CreativeML OpenRAIL-M
+  license further specifying rights and usage.
+
+  The CreativeML OpenRAIL License specifies: 
+
+
+  1. You can't use the model to deliberately produce nor share illegal or
+  harmful outputs or content 
+
+  2. The authors claim no rights on the outputs you generate, you are free to
+  use them and are accountable for their use which must not go against the
+  provisions set in the license
+
+  3. You may re-distribute the weights and use the model commercially and/or as
+  a service. If you do, please be aware you have to include the same use
+  restrictions as the ones in the license and share a copy of the CreativeML
+  OpenRAIL-M to all your users (please read the license entirely and carefully)
+
+  Please read the full license carefully here:
+  https://huggingface.co/spaces/CompVis/stable-diffusion-license
+      
+extra_gated_heading: Please read the LICENSE to access this model
+
+---
+
+
+# BK-SDM-v2 Model Card
+
+BK-SDM-{[**v2-Base**](https://huggingface.co/nota-ai/bk-sdm-v2-base), [**v2-Small**](https://huggingface.co/nota-ai/bk-sdm-v2-small), [**v2-Tiny**](https://huggingface.co/nota-ai/bk-sdm-v2-tiny)} are obtained by compressing [SD-v2.1-base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base). 
+- Block-removed Knowledge-distilled Stable Diffusion Models (BK-SDMs) are developed for efficient text-to-image (T2I) synthesis:
+  - Certain residual & attention blocks are eliminated from the U-Net of SD.
+  - Despite the use of very limited data, distillation retraining remains surprisingly effective.
+- Resources for more information: [Paper](https://arxiv.org/abs/2305.15798), [GitHub](https://github.com/Nota-NetsPresso/BK-SDM).
+
+
+## Examples with 🤗[Diffusers library](https://github.com/huggingface/diffusers).
+
+An inference code with the default PNDM scheduler and 50 denoising steps is as follows.
+
+```python
+import torch
+from diffusers import StableDiffusionPipeline
+
+pipe = StableDiffusionPipeline.from_pretrained("nota-ai/bk-sdm-v2-base", torch_dtype=torch.float16)
+pipe = pipe.to("cuda")
+
+prompt = "a black vase holding a bouquet of roses"
+image = pipe(prompt).images[0]  
+    
+image.save("example.png")
+```
+
+
+## Compression Method 
+
+Based on the [U-Net architecture](https://huggingface.co/nota-ai/bk-sdm-base#u-net-architecture) and [distillation retraining](https://huggingface.co/nota-ai/bk-sdm-base#distillation-pretraining) of BK-SDM, a reduced batch size (from 256 to 128) is used in BK-SDM-v2 for faster training speeds.
+
+- **Training Data**: 212,776 image-text pairs (i.e., 0.22M pairs) from [LAION-Aesthetics V2 6.5+](https://laion.ai/blog/laion-aesthetics/).
+- **Hardware:** A single NVIDIA A100 80GB GPU
+- **Gradient Accumulations**: 4
+- **Batch:** 128 (=4×32)
+- **Optimizer:** AdamW
+- **Learning Rate:** a constant learning rate of 5e-5 for 50K-iteration pretraining
+
+
+## Experimental Results 
+
+The following table shows the zero-shot results on 30K samples from the MS-COCO validation split. After generating 512×512 images with the PNDM scheduler and 25 denoising steps, we downsampled them to 256×256 for evaluating generation scores.
+
+- Our models were drawn at the 50K-th training iteration.
+
+#### Compression of SD-v2.1-base
+| Model | FID↓ | IS↑ | CLIP Score↑<br>(ViT-g/14) | # Params,<br>U-Net | # Params,<br>Whole SDM |
+|---|:---:|:---:|:---:|:---:|:---:|
+| [Stable Diffusion v2.1-base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base) | 13.93 | 35.93 | 0.3075 | 0.87B | 1.26B |
+| [BK-SDM-v2-Base](https://huggingface.co/nota-ai/bk-sdm-v2-base) (Ours) | 15.85 | 31.70 | 0.2868 | 0.59B | 0.98B |
+| [BK-SDM-v2-Small](https://huggingface.co/nota-ai/bk-sdm-v2-small) (Ours) | 16.61 | 31.73 | 0.2901 | 0.49B | 0.88B |
+| [BK-SDM-v2-Tiny](https://huggingface.co/nota-ai/bk-sdm-v2-tiny) (Ours) | 15.68 | 31.64 | 0.2897 | 0.33B | 0.72B |
+
+#### Compression of SD-v1.4
+| Model | FID↓ | IS↑ | CLIP Score↑<br>(ViT-g/14) | # Params,<br>U-Net | # Params,<br>Whole SDM |
+|---|:---:|:---:|:---:|:---:|:---:|
+| [Stable Diffusion v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4) | 13.05 | 36.76 | 0.2958 | 0.86B | 1.04B |
+| [BK-SDM-Base](https://huggingface.co/nota-ai/bk-sdm-base) (Ours) | 15.76 | 33.79 | 0.2878 | 0.58B | 0.76B |
+| [BK-SDM-Base-2M](https://huggingface.co/nota-ai/bk-sdm-base-2m) (Ours) | 14.81 | 34.17 | 0.2883 | 0.58B | 0.76B |
+| [BK-SDM-Small](https://huggingface.co/nota-ai/bk-sdm-small) (Ours) | 16.98 | 31.68 | 0.2677 | 0.49B | 0.66B |
+| [BK-SDM-Small-2M](https://huggingface.co/nota-ai/bk-sdm-small-2m) (Ours) | 17.05 | 33.10 | 0.2734 | 0.49B | 0.66B |
+| [BK-SDM-Tiny](https://huggingface.co/nota-ai/bk-sdm-tiny) (Ours) | 17.12 | 30.09 | 0.2653 | 0.33B | 0.50B |
+| [BK-SDM-Tiny-2M](https://huggingface.co/nota-ai/bk-sdm-tiny-2m) (Ours) | 17.53 | 31.32 | 0.2690 | 0.33B | 0.50B |
+
+#### Visual Analysis: Image Areas Affected By Each Word
+KD enables our models to mimic the SDM, yielding similar per-word attribution maps. The model without KD behaves differently, causing dissimilar maps and inaccurate generation (e.g., two sheep and unusual bird shapes).
+
+<center>
+    <img alt="cross-attn-maps" img src="https://netspresso-research-code-release.s3.us-east-2.amazonaws.com/assets-bk-sdm/fig_cross-attn-maps_bk-sd-v2.png" width="100%">
+</center>
+
+
+# Uses 
+Please follow [the usage guidelines of Stable Diffusion v1](https://huggingface.co/CompVis/stable-diffusion-v1-4#uses). 
+
+
+# Acknowledgments
+- [Microsoft for Startups Founders Hub](https://www.microsoft.com/en-us/startups) and [Gwangju AICA](http://www.aica-gj.kr/main.php) for generously providing GPU resources.
+- [CompVis](https://github.com/CompVis/latent-diffusion), [Runway](https://runwayml.com/), and [Stability AI](https://stability.ai/) for the pioneering research on Stable Diffusion.
+- [LAION](https://laion.ai/), [Diffusers](https://github.com/huggingface/diffusers), [PEFT](https://github.com/huggingface/peft), [DreamBooth](https://dreambooth.github.io/), [Gradio](https://www.gradio.app/), and [Core ML Stable Diffusion](https://github.com/apple/ml-stable-diffusion) for their valuable contributions.
+
+
+# Citation
+```bibtex
+@article{kim2023architectural,
+  title={BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion},
+  author={Kim, Bo-Kyeong and Song, Hyoung-Kyu and Castells, Thibault and Choi, Shinkook},
+  journal={arXiv preprint arXiv:2305.15798},
+  year={2023},
+  url={https://arxiv.org/abs/2305.15798}
+}
+```
+```bibtex
+@article{kim2023bksdm,
+  title={BK-SDM: Architecturally Compressed Stable Diffusion for Efficient Text-to-Image Generation},
+  author={Kim, Bo-Kyeong and Song, Hyoung-Kyu and Castells, Thibault and Choi, Shinkook},
+  journal={ICML Workshop on Efficient Systems for Foundation Models (ES-FoMo)},
+  year={2023},
+  url={https://openreview.net/forum?id=bOVydU0XKC}
+}
+```
+
+*This model card is based on the [Stable Diffusion v1 model card]( https://huggingface.co/CompVis/stable-diffusion-v1-4).*