“Flex Tape Can’t Fix That”: Bias and Misinformation in Edited Language Models (2024)

Abstract: Weight-based model editing methods update the parametric knowledge of language models post-training. However, these methods can unintentionally alter unrelated parametric knowledge representations, potentially increasing the risk of harm. In this work, we investigate how weight editing methods unexpectedly amplify model biases after edits. We introduce a novel benchmark dataset, Seesaw-CF, for measuring bias amplification of model editing methods for demographic traits such as race, geographic origin, and gender. We use Seesaw-CF to examine the impact of model editing on bias in five large language models. Our results demonstrate that edited models exhibit, to various degrees, more biased behavior for certain demographic groups than before they were edited, specifically becoming less confident in properties for Asian and African subjects. Additionally, editing facts about place of birth, country of citizenship, or gender has particularly negative effects on the model’s knowledge about unrelated properties, such as field of work, a pattern observed across multiple models.

## introduction due to the high cost of retraining language models, model editing methods have been proposed to update the knowledge encoded by models after deployment. branching out from variations on fine-tuning @xcite , researchers have developed various editing approaches, including editing model weights @xcite , using additional models with memory banks @xcite and decision rules @xcite , editing layer representations at run-time @xcite , and constructing demonstrative prompts @xcite . a principal challenge in model editing is to update a target fact and its logical corollaries without affecting other information that should remain the same. post-edit models are typically evaluated for specificity @xcite , which measures the extent to which the post-edit model modifies knowledge representations that are unrelated to the one targeted by the edit. however, specificity measurements penalize all unintended edits equally, overlooking the reality that certain alterations are potentially more harmful than others. one particularly problematic type of unintended alteration is one that exacerbate the model's existing bias toward subjects of certain demographic groups. models already are known to exhibit bias towards numerous social groups across various tasks @xcite @xcite . amplifying these biases could precipitate the generation of harmful rhetoric about those groups, which would be more harmful than merely mis-editing a singular fact. figure 1 shows such an example of an open-ended generation by gpt-j @xcite before and after being edited by the memit method @xcite , where the edit related to gender induces the model to subsequently produce a biased generation. to date, however, no studies have considered the potential unintended impact of model editing on the representations of certain demographic groups in models. in this work, we measure the downstream effects of model editing methods on model biases. we introduce seesaw-cf, a novel dataset for examining bias-related pitfalls of editing biographical facts in large language models (llm). seesaw-cf contains 3,516 knowledge edits across 5 properties (e.g., gender, field of work, citizenship) associated with human subjects, and measures the impact of applying these edits in three evaluation settings: cross-subject and cross-property cloze completion for bias assessment, and open-ended subject description. cross-subject completion evaluates a model's change in confidence about the same property for other subjects (e.g., does editing the birth place of a subject affect the model's confidence in the birth place of other subjects), which we stratify by different demographic groups. cross-property completion assesses the entanglement of biases among properties for the same individual (e.g., does editing a subject's gender affect the model's perception of their field of work?). open-ended subject description examines qualitative flaws and misinformation in extended model outputs (e.g., anglo-centrism, sexism, xenophobia, classism, racism, and religion and conservatism injection) after edits are applied, and is evaluated through both automated and human annotations to highlight more qualitative post-edit biases. we specifically investigate weight editing methods that risk undoing safety tuning, bias mitigation, and other critical adjustments. we focus on three methods: constrained fine-tuning (ft; @xcite , the direct editing method memit @xcite , and the hypernetwork-based method mend @xcite . we evaluate their effects on racial, geographic, and gender biases of autoregressive language models. we use gpt-j-6b @xcite , llama2-7b-hf @xcite , llama2-7b-chat-hf foot_0 , mistral-7b @xcite , and mistral-7binstruct foot_1 as editable models. to summarize, our contributions are: we release our code and data publicly. foot_2 considering the promise of model editing as an alternative to retraining, there has been an extensive exploration of its viability. overview works such as @xcite and @xcite provide systematic evaluations for an array of editing methods on the metrics of reliability, portability, generalization, and specificity (also referred to as locality; @xcite . reliability refers to the ability of an editing method to perform the desired edit, as measured by its average accuracy on facts that should be edited. generalization measures the propagation of an edit to semantically-equivalent expressions of the target fact, as measured by the post-edit model's accuracy on paraphrases in the equivalence neighborhood of the edited fact @xcite . specificity refers to an editing method's ability to keep information unchanged if it is unrelated to the edit, and is measured by a post-edit model's average accuracy on out-of-scope facts for a given edit. portability, a metric newly introduced by @xcite , measures a post-edit model's average accuracy across cases where (a) the subject of the fact is replaced with an alias or synonym, (b) the relation and subject are reversed in the phrasing, or (c) a model must reason about a logical corollary of the edited fact. the findings in these works highlight significant limitations in current model editing methods, particularly in terms of portability and specificity. when evaluating the quality of model editing methods, prior works have primarily measured edit success rate @xcite , specificity, and generalization @xcite , as well as the retention rate of original information @xcite , with some works beginning to look at the logical downstream implications of edited facts by examining multi-hop accuracy @xcite . for open-ended generation, some automatic metrics include consistency and fluency @xcite . fluency is measured both by human evaluation and by an automatic weighted average of bi-and tri-gram entropies of a generation, while consistency is measured as the cosine similarity between tfidf-vectorized forms of a generation and its corresponding reference texts sourced from wikipedia's descriptions of subjects sharing an edit object. @xcite introduce a benchmark for testing how editing affects model safety protocols, though they consider safety as a whole rather than examining group-specific safety concerns. however, researchers have yet to report these metrics disaggregated by demographic group or to investigate less automatically summarizable flaws in open-ended post-edit texts. our study aims to address both of these gaps, focusing on weightediting methods because they introduce more uncertainties and are less controllable than methods that solely build upon existing base models. ## seesaw-cf: a new dataset for bias detection in model editing methods in this work, we construct seesaw-cf, a dataset of 3,516 knowledge edits with a total of 734,620 accompanying cloze test prompts and 27,010 openended test prompts to facilitate the detection of bias-related pitfalls in model editing methods. each model edit in seesaw-cf edits a fact about a human subject and is accompanied by a set of prompts that measure the model's change in confidence for a collection of unrelated facts. prompt subjects are tagged with demographic traits, enabling measurement of bias across different groups. ## cloze completions we assess the impact of editing methods on cloze completions across multiple models, examining both cross-subject and cross-property scenarios. specifically, for editing methods, we apply ft, mend, and memit to gpt-j (the model for which these methods were initially designed). additionally, we use memit, identified as the most effective editing method (as discussed in section 5 and by @xcite ) to explore the effect of model editing on diverse models including llama 2, llama 2-chat, mistral, and mistral-instruct. ## open-ended descriptions in our cloze completion studies, we found that model editing amplified biases toward certain demographic groups ( §4.1), and changed unrelated property knowledge executing edits related to demographic categories ( §4.1). as diminished model confidence about entity subjects could significantly increase misinformation about those entities in open-ended generation tasks, we assess whether editing induces models to produce more biased descriptions in open-ended text generation. post-edit generations using unique prompts from counterfact, running each prompt five times for a total of 59,520 pairs. in these paired generations, we search for key flaws such as anglocentrism, sexism, religious injection, xenophobia, classism, racism, and conservatism injection. gpt-3.5 @xcite is used to score these pairs, indicating whether flaws are more present before or after the edit, or equally present in both. additionally, we conduct a human study with 252 randomly selected pairs generated by memit for gpt-j. 4 these pairs involve edits on citizenship (91 pairs), gender (74 pairs), and work (87 pairs), annotated by three us-based experts to determine the presence of flaws pre-and post-edit. detailed definitions of flaws, human annotation guidelines, and model prompts are provided in appendix f. ## discussion & conclusion this work introduces a novel dataset for biasrelated pitfalls in model editing and investigates demographic biases and qualitative flaws in text generation after editing model weights with ft, mend, and memit. our work is the first to specifically analyze the impact of model editing on demographic biases in llms. our findings show that model editing amplifies bias across all models and methods, albeit to varying degrees. in cross-subject scenarios, we find that the model's confidence in the attributes of certain demographic groups is more impacted by edits. for example, editing gender significantly reduces the model's confidence in the genders of asian, black, latine, middle eastern, and african subjects. in cross-property scenarios, we find that model representations of different knowledge properties are entangled, potentially allowing biases to propagate once edits are applied. for example, the field of work of many subjects is highly affected after editing the gender, birth, or citizenship of that subject. finally, qualitative assessments of open-ended descriptions of subject reveal increased levels of xenophobia, sexism, and the introduction of religious content post-edit. in terms of methods, fine-tuning (ft) and hypernetwork-based (mend) editing show increased susceptibility to biased factual bleedover, while direct editing (memit) escalates the generation of harmful texts. mend has the strongest effect on model confidence, with both ft and mend negatively influencing facts about language, citi-zenship, and birth. across models, the same demographic groups are affected, with the most bias amplification occurring after editing llama2, mistral, and mistral-instruct. overall, editing model weights carries significant risks of unintended bias and misinformation amplification. while biases in pre-trained models have been extensively studied, it is challenging to comprehensively evaluate these effects across all edited versions at scale. we provide seesaw-cf to the research community to specifically measure bias-related effects of editing.