Preference Tuning For Toxicity Mitigation Generalizes Across Languages (2024)

Abstract: Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their increasing global use. In this work, we explore zero-shot cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike previous studies that show limited cross-lingual generalization for other safety tasks, we demonstrate that Direct Preference Optimization (DPO) training with only English data can significantly reduce toxicity in multilingual open-ended generations. For example, the probability of mGPT-1.3B generating toxic continuations drops from 46.8% to 3.9% across 17 different languages after training. Our results also extend to other multilingual LLMs, such as BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal intervention and activation analysis, we identified the dual multilinguality property of MLP layers in LLMs, which explains the cross-lingual generalization of DPO. Finally, we show that bilingual sentence retrieval can predict the cross-lingual transferability of DPO preference tuning.

## introduction while significant resources have been allocated to enhance the safety of large language models (llms) for deployment, safety of multilingual llms remains underexplored @xcite . recent work has shown that multilingual llms have significant toxicity levels and therefore highlights the need for multilingual toxicity mitigation @xcite . however, to reduce toxicity in open-ended generations in a non-english language x, current solutions @xcite @xcite are resource-intensive as * equal contribution they require datasets of toxic and non-toxic samples in the language x, which is usually obtained through translating from english data @xcite due to resource unavailability. in this work, we study cross-lingual detoxification of llms using english preference tuning without translation. while prior work suggests limited cross-lingual transfer of preference tuning for the task of safeguarding against malicious instructions @xcite @xcite , we discover the opposite for llm detoxification task-we demonstrate zero-shot cross-lingual generalization of preference tuning in lowering toxicity of openended generations. specifically, we observe preference tuning with direct preference optimization (dpo) @xcite using only english training data can significantly reduce the toxicity level in llms' generations across 17 different languages, such as chinese, arabic, korean, russian and indonesian. our findings apply to multilingual llms of different sizes and with different pretraining composition, including mgpt @xcite , llama3 (ai@meta, 2024), and aya-23 @xcite . 1 we investigate the mechanisms enabling crosslingual generalization of safety preference tuning. recent work @xcite shows that models trained via dpo do not lose the ability to generate toxic content; instead, they learn to suppress the neuron activations that lead to toxicity, focusing on the role of key and value vectors in multi-layer perceptrons (mlp). while these findings explain dpo's effectiveness in the training language, they do not address its cross-lingual generalization. to bridge this gap, we extend the analysis to a multilingual context, and we demonstrate that both key a r c s d e e n e s f r h i i d i t j a k o n l p l p t r u s v z h 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 probability pre dpo post dpo (a) probability of generating toxic continuations a r c s d e e n e s f r h i i d i t j a k o n l p l p t r u s v z h 0.0 0.1 0.2 0.3 0.4 0.5 0.6 toxicity level pre dpo post dpo (b) expected maximum toxicity figure 1 : safety preference tuning on english (en) pairwise toxic/non-toxic data reduces mgpt's @xcite probability in generating toxic continuations (1a) and the expected toxicity level in its most-toxic generations (1b) across 17 different languages. we report results averaged over 5 seeds dpo training @xcite . our contributions can be summarized as below: ## related work cross-lingual generalization of rlhf/rlaif prior work suggests that zero-shot cross-lingual generalization of preference tuning with reinforcement learning with human feedback (rlhf) (or with ai feedback, rlaif) may be task-specific. for question-answering (qa), preference tuning of llms on english-dominant training data hurts its multilingual qa capability @xcite , and thus multilingual training data are needed @xcite . in contrast, for summarization, concurrent work demonstrates zeroshot cross-lingual generalization of rlhf with english reward models @xcite . similar findings apply to llm safety research. for the task of developing safeguards against malicious instructions, there is limited zero-shot crosslingual generalization to both low-resource @xcite @xcite and high-resource languages like chinese @xcite . here, we focus on another safety task, which is toxicity mitigation in open-ended generation @xcite . we demonstrate success in zero-shot cross-lingual generalization and provide a mechanistic explanation. jain et al. (2024) and de wynter et al. (2024) release multilingual toxicity evaluation benchmarks and they show that model toxicity increases as language resources decrease. to mitigate multilingual toxicity, current solutions @xcite require translating toxic and non-toxic data from english to target languages in order to extend existing detoxification methods @xcite to multilingual settings. @xcite also find limited zero-shot cross-lingual detoxification for supervised finetuning with models like m2m100 @xcite . in contrast, we demonstrate crosslingual detoxification with only english training data across different multilingual llms. ## cross-lingual toxicity mitigation we follow @xcite setup to perform preference tuning on llms for llm detoxification. specifically, we perform direct preference optimization (dpo) @xcite with @xcite preference dataset that consists of 24,576 instances of prompts as well as pairs of toxic (dispreferred) and non-toxic (preferred) continuations in english. we finetune five different base llms: (1) mgpt, a multilingual gpt with 1.3b parameters @xcite ; (2) bloom, a multilingual language model with 1.7b and 7.1b parameters (bigscience @xcite ; (3) aya-23, a multilingual language model with 8b parameters @xcite ; (4) llama2-7b @xcite ; and (5) llama3-8b (ai@meta, 2024). we perform full finetuning for mgpt and bloom-1.7b, and we use qlora adapters @xcite for finetuning models at 7b and 8b parameter sizes. we use huggingface trl library and follow @xcite hyperparameters (except learning rate) for full model finetuning of mgpt and bloom-1.7b. for qlora finetuning of aya-23, llama2, and llama3, we apply qlora @xcite on each model layer, with a rank of 64, a scaling parameter of 16 and a dropout of 0.05. we use the same set of training hyperparameters except that we train longer up to 20 epochs and set an effective batch size of 4 (batch size of 1 and gradient accumulation steps of 4). in all setups, we use early stopping by training until the validation loss input prompts across 17 different languages before and after english dpo preference tuning @xcite . ## mechanism in this section, we explain why english-only preference tuning can reduce toxicity in model generations across multiple languages using probes, causal intervention, and neuron activation analysis. ## predicting generalizability with bilingual sentence retrieval building upon our observations that the changes in activation levels differ across languages after dpo training (figure 3 ), we argue that the effectiveness of cross-lingual detoxification transfer from english to language x depends on how much english and x align in representations in the multilingual toxic subspace. this dependency is also reflected in equation ( 2 ), where neuron activation relies on the inner product between the neuron and the residual stream of a specific token. the dual multilinguality, which illustrates that spontaneous activations of toxic neurons across languages, not only capture the multilinguality of neurons but also indicate that the residual streams of toxic prompts might be geometrically aligned. the extent of this alignment can be approximated by bilingual sentence retrieval accuracy which is used to measure the quality of language-independent representations in prior work @xcite @xcite . bilingual sentence retrieval involves identifying semantically identical sentences in english based on a representation of the sentence in another language @xcite . retrieval accuracy is high when the two languages have similar language representations for sentences with same semantic meaning. we use 200 pairs of multiway parallel toxic prompts from rtp-lx dataset @xcite and obtain sentence representations for them at each layer of mgpt. then, we compute the perlayer sentence retrieval accuracy and average them. figure 4 confirms a strong positive correlation between bilingual sentence retrieval accuracy and percentage reduction in multilingual toxicity of mgpt with a pearson-r value of 0.73 (@xmath0.01). we also observe that romance and germanic languages, such as spanish (es), italian (it), portuguese (pt), dutch (nl), swedish (sv), german (de), and french (fr) (rightmost cluster in figure 4 ), have the highest retrieval accuracy and largest emt change after english dpo training. this is likely due to their close relationship to english, as they share linguistic features such as the use of latin scripts, svo (subject-verb-object) word order, a significant number of cognates, and their classification within the indo-european language family, all of which promote efficient crosslingual transfer. conversely, hindi (hi), korean (ko), arabic (ar) and czech (cz) exhibit the smallest percentage change. in addition to their language dissimilarity to english, these languages have the fewest training tokens for mgpt pretraining @xcite compared to the other 13 languages. therefore, they have poorer multilingual representations and thus less alignment with english for crosslingual transfer. we also observe similar findings for llama2-7b and bloom-7.1b (appendix e). our findings support previous work indicating that safety preference tuning has limited cross-lingual transfer for low-resource languages in pretraining @xcite . ## conclusion we show that safety preference tuning with dpo to detoxify llms can generalize across languages in a zero-shot manner. our findings are robust to different multilingual llms. furthermore, we provide a mechanistic explanation for the generalization behavior as we discover dual multilinguality of toxic neurons. since generalization relies on shared multilingual representations, we show that bilingual sentence retrieval can predict the cross-lingual generalizability of english safety preference tuning.