concept ablation

README.md

@@ -1,5 +1,5 @@
 ---
-title: Erasing Concepts from Diffusion Models 
+title: Ablating Concepts in Text-to-Image Diffusion Models 
 emoji: 💡
 colorFrom: indigo
 colorTo: gray
@@ -12,24 +12,21 @@ license: mit
 
 
 
-# Erasing Concepts from Diffusion Models
+# Ablating Concepts in Text-to-Image Diffusion Models
 
- Project Website    [https://erasing.baulab.info](https://erasing.baulab.info) <br>
- Arxiv Preprint     [https://arxiv.org/pdf/2303.07345.pdf](https://arxiv.org/pdf/2303.07345.pdf) <br>
- Fine-tuned Weights [https://erasing.baulab.info/weights/esd_models/](https://erasing.baulab.info/weights/esd_models/) <br>
+ Project Website    [https://www.cs.cmu.edu/~concept-ablation/](https://www.cs.cmu.edu/~concept-ablation/) <br>
+ Arxiv Preprint     [https://arxiv.org/abs/2303.13516](https://arxiv.org/abs/2303.13516) <br>
 <div align='center'>
 <img src = 'images/applications.png'>
 </div>
 
-Motivated by recent advancements in text-to-image diffusion, we study erasure of specific concepts from the model's weights. While Stable Diffusion has shown promise in producing explicit or realistic artwork, it has raised concerns regarding its potential for misuse. We propose a fine-tuning method that can erase a visual concept from a pre-trained diffusion model, given only the name of the style and using negative guidance as a teacher. We benchmark our method against previous approaches that remove sexually explicit content and demonstrate its effectiveness, performing on par with Safe Latent Diffusion and censored training.
+Large-scale text-to-image diffusion models can generate high-fidelity images with powerful compositional ability. However, these models are typically trained on an enormous amount of Internet data, often containing copyrighted material, licensed images, and personal photos. Furthermore, they have been found to replicate the style of various living artists or memorize exact training samples. How can we remove such copyrighted concepts or images without retraining the model from scratch?
 
-To evaluate artistic style removal, we conduct experiments erasing five modern artists from the network and conduct a user study to assess the human perception of the removed styles. Unlike previous methods, our approach can remove concepts from a diffusion model permanently rather than modifying the output at the inference time, so it cannot be circumvented even if a user has access to model weights
-
-Given only a short text description of an undesired visual concept and no additional data, our method fine-tunes model weights to erase the targeted concept. Our method can avoid NSFW content, stop imitation of a specific artist's style, or even erase a whole object class from model output, while preserving the model's behavior and capabilities on other topics.
+We propose an efficient method of ablating concepts in the pretrained model, i.e., preventing the generation of a target concept. Our algorithm learns to match the image distribution for a given target style, instance, or text prompt we wish to ablate to the distribution corresponding to an anchor concept, e.g., Grumpy Cat to Cats.
 
 ## Demo vs github
 
-This demo uses an updated implementation from the original Erasing codebase the publication is based from.
+This demo uses different hyper-parameters than the github version for faster training.
 
 ## Running locally
 
@@ -39,15 +36,15 @@ This demo uses an updated implementation from the original Erasing codebase the
 
 3.) Open the application in browser at `http://127.0.0.1:7860/`
 
-4.) Train, evaluate, and save models using our method
+4.) Train, evaluate, and save models
 
 ## Citing our work
 The preprint can be cited as follows
 ```
-@article{gandikota2023erasing,
-  title={Erasing Concepts from Diffusion Models},
-  author={Rohit Gandikota and Joanna Materzy\'nska and Jaden Fiotto-Kaufman and David Bau},
-  journal={arXiv preprint arXiv:2303.07345},
-  year={2023}
+@inproceedings{kumari2023conceptablation,
+  author = {Kumari, Nupur and Zhang, Bingliang and Wang, Sheng-Yu and Shechtman, Eli and Zhang, Richard and Zhu, Jun-Yan},
+  title = {Ablating Concepts in Text-to-Image Diffusion Models},
+  booktitle = ICCV,
+  year = {2023},
 }
 ```