Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models (2024)

Abstract: Pretrained language models memorize vast amounts of information, including private and copyrighted data, raising significant safety concerns. Retraining these models after excluding sensitive data is prohibitively expensive, making machine unlearning a viable, cost-effective alternative. Previous research has focused on machine unlearning for monolingual models, but we find that unlearning in one language does not necessarily transfer to others. This vulnerability makes models susceptible to low-resource language attacks, where sensitive information remains accessible in less dominant languages. This paper presents a pioneering approach to machine unlearning for multilingual language models, selectively erasing information across different languages while maintaining overall performance. Specifically, our method employs an adaptive unlearning scheme that assigns language-dependent weights to address different language performances of multilingual language models. Empirical results demonstrate the effectiveness of our framework compared to existing unlearning baselines, setting a new standard for secure and adaptable multilingual language models.

## introduction privacy regulations such as the right to be forgotten (rtbf) @xcite , the european union's general data protection regulation (gdpr) @xcite , and the united states' california consumer privacy act (ccpa) @xcite mandate that individuals have the right to request the deletion of their data from databases, which extends to data held within machine learning (ml) models. additionally, the writers guild of america strike in 2023 highlighted increasing concerns regarding the copyrighted content generated by large language models (llms) @xcite . consequently, removing such information in just one language does not entirely eradicate it from the model. this underscores the necessity for a multilingual unlearning approach to ensure the information is thoroughly eliminated from the model. to comply with such issues, significant attention has been directed towards the task of machine unlearning (mu), which involves removing the influence of specific data points from ml models @xcite . despite the critical necessity of the task, mitigating the influence of data samples on billions of model parameters presents an immense challenge. the most definitive method is exact unlearning, which necessitates retraining ml models entirely from scratch, utilizing the residual training dataset after excising the specified data points. however, this method is computationally prohibitive and not feasible, particularly for llms. therefore, the advancement of rapid approximate unlearning methodologies has emerged as a primary focus of contemporary research efforts. research on mu has predominantly focused on computer vision tasks @xcite @xcite ; however, it is now gaining traction in nlp due to the safety issues that arise with llms @xcite . notably, @xcite first proposed an unlearning technique of reversing the gradient to refrain lms from generating particular sensitive token sequences. on the other hand, @xcite presented an approach to maintaining distribution differences (i.e., knowledge gap) such that the performance of the data to be forgotten becomes similar to the performance of the unseen data. besides the two approaches, substantial progress has been made in unlearning for monolingual models; nevertheless, there is a lack of empirical results and analyses of unlearning for multilingual lms. as shown in figures 1 and 2 , our preliminary experiments find that existing unlearning approaches do not exhibit cross-lingual transferability. in other words, unlearning in one language does not automatically transfer to other languages, leaving lms vulnerable to possible lowresource language attacks, which have been shown to jailbreak @xcite . to this end, we introduce multilingual unlearning, which effectively removes specific information across a wide variety of languages from pretrained language models. 2 due to the inconsistency in model performance across languages, we leverage a multilingual teacher model in which the student model adaptively obeys the teacher based on its capabilities in a particular language. for example, a high knowledge distillation weight is applied when the teacher has strong expertise, ensuring the benefit of effective teaching. conversely, a low weight 2 although the task can be applied to monolingual models that may have some multilingual capabilities, we focus on multilingual lms to limit the scope of our study. to assess the success of unlearning across different languages, our experimental setup necessitates multilingual parallel data. however, obtaining such datasets is challenging, especially when dealing with a particular domain. consequently, we evaluate our framework using two multilingual parallel datasets in the general domain, which are utilized to unlearn specific token sequences and factual knowledge across various languages, respectively. empirical results demonstrate that our proposed approach surpasses existing unlearning methods by a considerable margin. given the intrinsic similarities in unlearning token sequences, we believe these datasets provide an appropriate testbed for evaluating multilingual unlearning. overall, the major contributions of our work are as follows: ## problem definition knowledge unlearning given a token sequence @xmath0, the task of knowledge unlearning is to safely remove the influence of a subset of data d f from a trained machine learning model such that the model behaves as if the removed data had never been part of the training process, thus maintaining the model performance for the rest of the dataset. conventionally, the data to be forgotten d f is expressed as the forget set, while the data to be retained d r is named as the retain set. for simplicity, we consider the standard case where d f and d r represent the whole training dataset and are mutually exclusive; that is, d f ∪ d @xmath1. the objective is to adjust the model parameters θ such that the updated parameters θ ′ = s(θ; d f ) reflect the removal of d f . this unlearning (scrubbing) function s ensures the model behaves as if trained solely on d r , effectively forgetting d f while maintaining performance on d r . multilingual unlearning extending to a multilingual context, the above definition must hold for the dataset d across all possible languages. while ideally, unlearning should occur across all existing languages, our experiments focus on a predefined set of languages @xmath2. consequently, we assume a parallel dataset with forget sets ## conclusion in response to rising privacy concerns and regulatory demands, our study pioneers a method for machine unlearning in multilingual language models. we introduce an adaptive unlearning scheme using a multilingual teacher model to address performance disparities across languages, ensuring the effective removal of sensitive information while maintaining overall model performance. our empirical results, validated on multilingual parallel datasets, demonstrate significant improvements over existing unlearning methods. this approach not only mitigates vulnerabilities to low-resource language attacks but also offers a practical, efficient alternative to retraining models from scratch, aligning with modern privacy regulations and advancing the field of nlp.