Ethos: Rectifying Language Models in Orthogonal Parameter Space (2024)

Abstract: Language models (LMs) have greatly propelled the research on natural language processing. However, LMs also raise concerns regarding the generation of biased or toxic content and the potential disclosure of private information from the training dataset. In this work, we present a new efficient approach, Ethos, that rectifies LMs to mitigate toxicity and bias in outputs and avoid privacy leakage. Ethos is built on task arithmetic. However, unlike current task arithmetic algorithms, Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors. Specifically, Ethos first obtains a set of principal components from the pre-trained models using singular value decomposition. Then, by projecting the task vector onto principal components, Ethos separates the principal components that encode general from those associated with undesired knowledge. Ethos performs forgetting or unlearning by only negating the task vector with undesired knowledge, thereby minimizing collateral damage on general model utility. We demonstrate the efficacy of our approach on three different tasks: bias, toxicity, and memorization unlearning. Evaluations show Ethos is more effective in removing undesired knowledge while maintaining the overall model performance compared to current task arithmetic methods.

## introduction the advent of language models (lms) has enhanced the current capabilities in text understanding and generation @xcite @xcite . due to their significant potential lms have been the driving force in many automated systems that improve productivity in real-world tasks @xcite @xcite . however, despite their success, lms also bring to * these authors contributed equally. the forefront some new challenges. this paper focuses on one pivotal challenge among these: lms' propensity to generate toxic, biased content or reveal private training records. overview of toxicity/bias/privacy concerns of lms: since lms are pre-trained with a large volume of data, the composition of the dataset during pre-training can greatly affect the performance of lms. in particular, suppose a dataset used in pre-training contains a substantial amount of toxic information, it can result in an lm that is likely to generate toxic or harmful messages for certain prompts @xcite . similarly, an imbalanced dataset with unevenly distributed data points among groups (e.g., gender, race, ethnicity) can lead to the development of biases in lms @xcite @xcite . for instance, lms may associate certain features with a gender group when pre-trained on gender-imbalanced datasets. another critical concern in deploying lms is the risk of privacy leakage due to model memorization. specifically, lms tend to overfit training data and memorize specific examples, increasing vulnerability to privacy breaches, such as training data extraction attacks @xcite @xcite . memorization compromises privacy and poses security risks, especially when the training data contains sensitive information. addressing these challenges is crucial in the development of lms. a naive approach is to retrain the model from scratch, for instance, whenever bias or memorization is discovered and removed from the training data. considering the prohibitive costs of training lms, it is infeasible to re-train the model. hence, the objective of this work is to rectify lms without incurring substantial costs. overview of model editing by task arithmetic. prior work @xcite introduces a model editing method that reduces toxic informa-tion in outputs by directly editing models with a task vector. the task vector, obtained after finetuning the model on a downstream dataset, encodes certain undesired knowledge (e.g., toxicity). therefore, negating such a task vector helps rectify lms and forgetting or unlearning undesired bias while maintaining reasonable model performance. to further improve the model editing performance, zhang et al. leverage parameter-efficient fine-tuning methods such as low-rank adaptation (lora) @xcite to edit the task vector formed by a subset of the model weights using parameter-efficient modules only rather than the full model weights. current model editing methods still struggle to maintain lms' performance when directly operating in the parameter space. the reason is that task vectors mix undesired knowledge with the general knowledge that is necessary for preserving model utility @xcite . as a result, simply negating the task vector on an lm inevitably removes the general knowledge alongside the undesired knowledge, causing collateral damage to the overall model performance. we present more detailed related work in appendix a. ## methodology the objective of this study is to edit lms to remove certain types of undesired knowledge encoded in lms, such as bias, toxicity, or certain private information. existing methods that rely on task vectors are unable to distinguish undesired knowledge from overall beneficial knowledge within those vectors. therefore, we propose ethos that is aimed to remove only undesired knowledge and ensure the edited model is rectified without significantly compromising the model utility. next, we present our method, ethos. at a high level, ethos decomposes weights of a pre-trained model along orthogonal directions and analyzes changes in each direction when fine-tuning the pretrained model on a downstream task. we demonstrate that each direction represents a specific type of knowledge that is orthogonal to the others. during fine-tuning, directions with general knowledge that exist in the pre-trained model will observe marginal changes, while substantial changes can happen along directions with task-specific knowledge. therefore, ethos constructs a new task vector only along these task-specific directions and negates the task vector on the pre-trained model. hence, with a proper downstream dataset, one can identify orthogonal spaces that are most impacted by bias or toxic information. as shown in figure 1 , ethos consists of the following key steps. task alignment. given a pre-trained model, θ pt , we first align it with the downstream task. since the pre-trained model lacks knowledge about the downstream task, the alignment step is necessary for constructing an orthogonal space that captures the downstream context. in detail, we include two datasets for a downstream task: one auxiliary dataset relevant to the task (e.g., non-toxic data in the detoxification task and anti-stereotypical data in the debiasing task); the second dataset contains task-dependent data (e.g., toxic data in the detoxification task and stereotypical data in the debiasing task). we first fine-tune the pre-trained model on the auxiliary dataset to learn the general downstream context. we denote the fine-tuned model as θ ′ @xmath0. knowledge separation. as stated in prior works @xcite , θ ′ pt implicitly memorizes knowledge from training datasets, including general and undesired knowledge. the key first step in our method is to construct a separable space so that we can project weights onto separable directions and analyze the role of each direction. inspired by orthogonality in linear algebra and its applications in machine learning @xcite , we say w 1 and w 2 encode orthogonal knowledge if w * 1 • w 2 = 0. we can understand the definition via a linear layer in lms. for a linear layer, given input x, output after w 1 and w 2 is we can see that if w 1 is orthogonal to w 2 , their outputs are also orthogonal. specifically, therefore, given input x, outputs after w 1 and w 2 contain information that is orthogonal. with the observation above, we can convert the pre-trained model, θ ′ pt , into an orthogonal space, where each direction can denote knowledge that is orthogonal to other directions. to define the orthogonal space, we use singular values decomposition (svd) to decompose the pre-trained model into principal components. given weights in i-th layer, @xmath1, we decompose it as to all others, the output after w k also represents orthogonal information to outputs from other principal components. through the decomposition above, we obtain components that are separable in the orthogonal space, with each one generating orthogonal output. ## empirical evaluations in this section, we conduct evaluations of ethos on various unlearning tasks, detailing the evaluation setup and discussing the results for each task. ## discussion in this section, we analyze the necessity of the auxiliary task vector when performing a projection 2061 in ethos. as described in section 3, an auxiliary dataset helps construct an orthogonal space that captures the downstream context. therefore, the initial task vector on the downstream task, ∆θ task , and the model θ ′ pt , are more aligned in the orthogonal space. as stated in section 3, with the aligned orthogonal components in ∆θ task and θ ′ pt , less errors are introduced during projection in eq ( 5) . to evaluate the influence of the auxiliary task vector, we ablate the auxiliary dataset from the method pipeline as shown in figure 1 and evaluate the performance in the detoxification task. as indicated in the auxiliary dataset enables the pre-trained model to learn the downstream instruction format and context, rather than capturing all information present in the pre-trained dataset. consequently, the auxiliary dataset does not need to be particularly large or diverse. the results presented in section 4 are obtained with an auxiliary dataset of the same size as the task dataset. for instance, for the detoxification task discussed in section 4.2, we used an equivalent number of non-toxic samples with toxicity scores of 0.0 from the civil comments dataset, approximately 23,000 samples. similarly, for the debiasing task in section 4.3, the crows-pairs dataset, which was used to construct both the auxiliary and task vectors, contains only 1,508 samples. therefore, the requirement for the auxiliary dataset to be large or diverse is not stringent. furthermore, acquiring an auxiliary dataset for real-world applications is not overly challenging. specifically, for potential future tasks like untruthfulness unlearning, the auxiliary dataset can easily be constructed using a text corpus with truthful information, such as public datasets like truthfulqa @xcite . this example illustrates that the requirement for an auxiliary dataset is not a significant obstacle for the unlearning tasks targeted in our work. ## conclusion this paper introduces a novel and efficient method for rectifying lms and addresses the critical issues of toxicity, bias, and privacy leaks. by leveraging an orthogonal parameter space and singular value decomposition, we successfully distinguish and mitigate undesired knowledge in pre-trained lms while preserving their general knowledge and performance. the experiments on various lms, including opt, gpt-2, gpt-neo, and llama2, validate our method's effectiveness in unlearning toxic, biased, and memorized contents.