add model card

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 rezashkv
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -2,6 +2,116 @@
 license: mit
 language:
 - en
-library_name: diffusers
-pipeline_tag: text-to-image
----
+
+tags:
+- text-to-image
+- stable-diffusion
+- diffusers
+---
+
+
+# APTP: Adaptive Prompt-Tailored Pruning of T2I Diffusion Models
+[![arXiv](https://img.shields.io/badge/Paper-arXiv-red?style=for-the-badge)]() 
+[![Github](https://img.shields.io/badge/Gihub-Code-succees?style=for-the-badge&logo=GitHub)](https://github.com/rezashkv/diffusion_pruning)
+
+The implementation of the paper ["Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models"](https://openreview.net/forum?id=ekR510QsYF)
+
+## Abstract
+
+Text-to-image (T2I) diffusion models have demonstrated impressive image generation capabilities. Still, their computational intensity prohibits 
+resource-constrained organizations from deploying T2I models after fine-tuning them on their internal target data. While pruning 
+techniques offer a potential solution to reduce the computational burden of T2I models, static pruning methods use the same pruned 
+model for all input prompts, overlooking the varying capacity requirements of different prompts. Dynamic pruning addresses this issue by utilizing 
+a separate sub-network for each prompt, but it prevents batch parallelism on GPUs. To overcome these limitations, we introduce 
+Adaptive Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed for T2I diffusion models. Central to our approach is a 
+prompt router model, which learns to determine the required capacity for an input text prompt and routes it to an architecture code, given a 
+total desired compute budget for prompts. Each architecture code represents a specialized model tailored to the prompts assigned to it, and the 
+number of codes is a hyperparameter. We train the prompt router and architecture codes using contrastive learning, ensuring that similar prompts 
+are mapped to nearby codes. Further, we employ optimal transport to prevent the codes from collapsing into a single one. We demonstrate APTP's 
+effectiveness by pruning Stable Diffusion (SD) V2.1 using CC3M and COCO as target datasets. APTP outperforms the 
+single-model pruning baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters learned by APTP reveals they 
+are semantically meaningful. We also show that APTP can automatically discover previously empirically found challenging prompts for SD, e.g., prompts for generating text images, assigning them to higher capacity codes.
+
+
+<p align="center">
+  <img src="assets/fig_1.gif" alt="APTP Overview" width="600" />
+</p>
+<p align="left">
+  <em>APTP: We prune a text-to-image diffusion model like Stable Diffusion (left) into a mixture of efficient experts (right) in a prompt-based manner. Our prompt router routes distinct types of prompts to different experts, allowing experts' architectures to be separately specialized by removing layers or channels.</em>
+</p>
+
+<p align="center">
+  <img src="assets/fig_2.gif" alt="APTP Pruning Scheme" width="600" />
+</p>
+<p align="left">
+  <em>APTP pruning scheme. We train the prompt router and the set of architecture codes to prune a T2I diffusion model into a mixture of experts. The prompt router consists of three modules. We use a Sentence Transformer as the prompt encoder to encode the input prompt into a representation z. Then, the architecture predictor transforms z into the architecture embedding e that has the same dimensionality as architecture codes. Finally, the router routes the embedding e into an architecture code a(i). We use optimal transport to evenly distribute the prompts in a training batch among the architecture codes. The architecture code a(i) = (u(i), v(i)) determines pruning the model’s width and depth. We train the prompt router’s parameters and architecture codes in an end-to-end manner using the denoising objective of the pruned model L<sub>DDPM</sub>, distillation loss between the pruned and original models L<sub>distill</sub>, average resource usage for the samples in the batch R, and contrastive objective L<sub>cont</sub>, encouraging embeddings e preserving semantic similarity of the representations z.</em>
+</p>
+
+
+### Model Description
+
+- **Developed by:** UMD Heng Lab
+- **Model type:** Text-to-Image Diffusion Model
+- **Model Description:** APTP is a pruning scheme for text-to-image diffusion models like Stable Diffusion, resulting in a mixture of efficient experts specialized for different prompt types.
+
+### License
+
+APTP is released under the MIT License. Please see the [LICENSE](LICENSE) file for details.
+
+### Model Sources
+
+For local or self-hosted use, follow the instructions in the [Github Repository](https://github.com/rezashkv/diffusion_pruning)
+
+
+## Training Dataset
+
+We used Conceptual Captions and MS-COCO 2014 datasets for training the models. Details for downloading and preparing these datasets are provided in the [Github Repository](https://github.com/rezashkv/diffusion_pruning).
+
+## File Structure
+
+```
+APTP
+├── APTP-Base-CC3M
+│   ├── arch0  
+│   ├── ...
+│   └── arch15
+├── APTP-Small-CC3M
+│   ├── arch0
+│   ├── ...
+│   └── arch7
+├── APTP-Base-COCO
+│   ├── arch0
+│   ├── ...
+│   └── arch7
+└── APTP-Small-COCO
+    ├── arch0
+    ├── ...
+    └── arch7*
+```
+
+
+## Uses
+
+This model is designed for academic and research purposes, specifically for exploring the efficiency of text-to-image diffusion models through prompt-based pruning. Potential applications include:
+
+1. **Research:** Researchers can use the model to study prompt-based pruning techniques and their impact on the performance and efficiency of text-to-image generation models.
+2. **Education:** Educators and students can use this model as a learning tool for understanding advanced concepts in neural network pruning, diffusion models, and prompt engineering.
+3. **Benchmarking:** The model can be used for benchmarking against other text-to-image generation models to assess the trade-offs between computational efficiency and output quality.
+
+
+## Safety
+
+When using these models, it is important to consider the following safety and ethical guidelines:
+
+1. **Content Generation:** The model can generate a wide range of images based on text prompts. Users should ensure that the generated content adheres to ethical guidelines and does not produce harmful, offensive, or inappropriate images.
+2. **Bias and Fairness:** Like other AI models, APTP may exhibit biases present in the training data. Users should be aware of these potential biases and take steps to mitigate their impact, particularly when the model is used in sensitive or critical applications.
+3. **Data Privacy:** Ensure that any data used with the model complies with data privacy regulations. Avoid using personally identifiable information (PII) or sensitive data without proper consent.
+4**Responsible Use:** Users are encouraged to use the model responsibly, considering the potential social and ethical implications of their work. This includes avoiding the generation of misleading or false information and respecting the rights and dignity of individuals depicted in generated images.
+
+By adhering to these guidelines, users can help ensure the responsible and ethical use of the APTP model.
+
+## Contact
+In case of any questions or issues, please contact the authors of the paper:
+
+* [Reza Shirkavand](mailto:rezashkv@umd.edu)
+* [Alireza Ganjdanesh](mailto:aliganj@umd.edu)