Side Effects of Erasing Concepts from Diffusion Models (2025)

Abstract: Concerns about text-to-image (T2I) generative models infringing on privacy, copyright, and safety have led to the development of concept erasure techniques (CETs). The goal of an effective CET is to prohibit the generation of undesired “target” concepts specified by the user, while preserving the ability to synthesize high-quality images of other concepts. In this work, we demonstrate that concept erasure has side effects and CETs can be easily circumvented. For a comprehensive measurement of the robustness of CETs, we present the Side Effect Evaluation (SEE) benchmark that consists of hierarchical and compositional prompts describing objects and their attributes. The dataset and an automated evaluation pipeline quantify side effects of CETs across three aspects: impact on neighboring concepts, evasion of targets, and attribute leakage. Our experiments reveal that CETs can be circumvented by using superclass-subclass hierarchy, semantically similar prompts, and compositional variants of the target. We show that CETs suffer from attribute leakage and a counterintuitive phenomenon of attention concentration or dispersal. We release our benchmark and evaluation tools to aid future work on robust concept erasure.

## introduction text-to-image (t2i) diffusion models generate images based on text prompts @xcite , harnessing the expressive power of natural language to create new images. although t2i models generate photorealistic images, they pose the risk of generating images that contain harmful @xcite and copyright-protected @xcite content as they are trained on large-scale online data. the task of concept erasure has emerged as a solution to this, aiming to remove undesired target concepts from the knowledge of pre-trained models while preserving other capabilities. while there is much impetus to develop such concept erasure techniques (cets), there is a gap in understanding the ability of these methods to safely remove a specific concept without degrading the ability to generate images of other concepts, which need to be preserved. in this work, we pursue the question: to what extent can cets remove a target concept without introducing unintended side effects in t2i models? figure 1 shows images generated by a state-of-theart cet for prompts with objects and associated attributes, illustrating three types of side-effects that we study in this paper: impact on neighboring concepts, evasion of erasure, and attribute leakage. our work highlights that existing evaluation metrics for concept erasure fail to identify these side effects, resulting in an incomplete picture of challenges in this task. this finding highlights the need for a dedicated benchmark to systematically quantify capabilities and limitations of cets. we develop the see dataset that contains compositional text prompts describing objects (e.g. "chair") and attributes (e.g."small red metallic chair"). see contains 5056 compositional prompts, built on commonly occurring ms-coco @xcite objects categorized into 11 superclasses. we develop an automated evaluation pipeline that leverages this dataset to conduct a large-scale evaluation of the side effects of cets. using this approach, we evaluate six state-of-the-art cet methods: uce @xcite , rece @xcite , mace @xcite , spm @xcite , esd @xcite , and advunlearn @xcite applied to the stable diffusion @xcite t2i model. for each cet, we generate and evaluate four images per prompt, resulting in a large-scale evaluation of 20,224 images per model. our experiments reveal several vulnerabilities of cets. first, we find that all of the cets fail to erase : vehicle expectation: no vehicle reality: vehicle side effect: evasion unedited edited prompt an image of a car an image of a red car expectation: no car reality: red car erase : car vehicle car bus red car small car …. blue car side effect: child not erased an image of a red bus erase : red car vehicle car bus red car small car …. blue car expectation: red bus reality: white bus side effect: impact on neighbor erase : couch expectation: no couch reality: no couch, two blue potted plants side effect: attribute leakage an image of a blue couch and a potted plant. vehicle car bus red car small car …. blue car furniture couch potted plant blue couch blue potted plant large couch … red potted plant figure 1 : we benchmark unintended side effects of cets. each column shows the concept to be erased, the text prompt, and the images generated before (top) and after (bottom) erasure. the tree shows the sub-graph in the hierarchy (parents and children) corresponding to the erased concept. we highlight the side effects: (1) impact on neighboring concepts: erasing "car" does not erase the child concept "red car", while erasing "red car" impacts the neighboring concept red bus. ## conclusion this work introduces see, a large-scale automated benchmark for comprehensive evaluation of concept erasure in t2i diffusion models. previous evaluations have relied on testing only target concepts; for instance, when erasing "car", only the model's ability to generate cars is tested. we demonstrate this approach is inadequate and that evaluation should encompass related sub-concepts like "red uce rece mace spm erase: red couch, large couch, … all-at-once progressive prompt an image of a couch uce rece mace spm erase: red teddy bear, large teddy bear, … all-at-once progressive prompt an image of a teddy bear figure 12: comparison between progressive vs. allat-once erasure strategies. for both target concepts "couch" and "teddy bear", when the entire erase set e is erased all-at-once, the edited models continue to generate couch and bear-like objects. however, when concepts from e are erased progressively, edited models behave more effectively: although rece and mace produce couch-like objects (top row), none of them generate a teddy bear (bottom row). limitations. while we focus on three major side effects, the failure modes uncovered in our analysis suggest that additional side effects of concept erasure may exist and warrant further investigation. this work initiates research on robust evaluation of concept erasure techniques to spark further work in this direction. in this benchmark, "concepts" are restricted to object categories and supercategories, and only verifiable attributes such as size, color, and material are used such that visual recognition models can automatically detect them. the benchmark can be extended to more attributes when more sophisticated recognition techniques may emerge for those attributes. finally, our study focuses on cets that adopt closed-form solutions, which are more practical to deploy due to their efficiency and minimal computational overhead. however, this excludes finetuning-based cets, which may exhibit distinct side effects that are not captured by our current evaluation. umbc's strategic award for research transitions (start). mg was partially supported with umbc cyberscurity award. high performance computing support was provided by umbc hpcf.