The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (2024)

Abstract: A key concern with the concept of *“alignment”* is the implicit question of *“alignment to what?”*. AI systems are increasingly used across the world, yet safety alignment is often focused on homogeneous monolingual settings. Additionally, preference training and safety measures often overfit to harms common in Western-centric datasets. Here, we explore the viability of different alignment approaches when balancing dual objectives: addressing and optimizing for a non-homogeneous set of languages and cultural preferences while minimizing both global and local harms. We collect the first human annotated red teaming prompts in different languages, distinguishing between global and local harm, which serve as a laboratory to understand the reliability of alignment techniques when faced with preference distributions that are non-stationary across geographies and languages. While this setting is seldom covered by the literature to date, which primarily centers on English harm mitigation, it captures real-world interactions with AI systems around the world. We establish a new precedent for state-of-the-art alignment techniques across 6 languages with minimal degradation in general performance. Our work provides important insights into cross-lingual transfer and novel optimization approaches to safeguard AI systems designed to serve global populations.

## introduction recent progress and developments in the field of natural language processing (nlp) have resulted in increasingly impressive capabilities of large language models (llms) across a variety of complex domains and tasks such as multi-task generalization @xcite , reasoning @xcite and in-context learning @xcite to name a few. however, these improved abilities are accompanied with risks and concerns around alignment and safety @xcite @xcite @xcite . many works show that llms can be maliciously prompted to elicit harmful behaviors which can have deleterious effects @xcite @xcite @xcite . numerous strategies, ranging from technical solutions @xcite @xcite @xcite to ethical frameworks @xcite have been developed to tackle the potential harms of llms. however, a common thread among the vast majority of work on safety and alignment is the narrow focus on addressing safety exclusively for a monolingual setting: english. given the widespread adoption of llms around the world, this creates a sharp cliff in performance which disproportionately amplifies risk for non-english speakers @xcite @xcite @xcite @xcite . furthermore, it can introduce critical security and safety flaws for all users as languages outside of english are often considered out-of-distribution and can be used to subvert safety guardrails @xcite @xcite @xcite . improving alignment across languages is a formidable challenge. this includes addressing the pronounced problem of multilingual data scarcity @xcite , especially of preference training datasets. languages are deeply rooted in the cultural and social fabric of a community and they evolve to capture the unique nuances and perspectives of groups of people @xcite @xcite . while automatic translation is a popular starting point to address data scarcity @xcite , relying solely on translations is not sufficient as it can introduce artefacts, and nuances of the original script can be hard to capture in translation @xcite @xcite . successful mitigation of multilingual harm also involves reconciling differing global and local preferences. this involves grappling with some of the core tensions that characterize machine learning: optimizing for a non-homogeneous distribution with evolving dynamics across languages and geographies. the vast majority of work to-date focused on alignment, including works on reinforcement learning from human feedback (rlhf) @xcite @xcite or even ai feedback (rlaif) @xcite , has focused on monolingual settings. in this work, we aim to push the limits of aligning language models by optimizing for general and safety performance simultaneously in a multilingual setting. our work seeks to answer key questions such as 1) must mitigation techniques be language specific to be effective? 2) what are the cross-lingual benefits of alignment techniques? 3) should the mitigation for "local" and "global" harms be tackled separately? we conduct extensive experiments across several mitigation techniques, from supervised fine-tuning (sft) to the more recently popularized direct preference opti-mization (dpo) @xcite , to measure the trade-offs of different methods. we conduct extensive llm-as-a-judge evaluations combined with human evaluation to characterize the cross-over between global and local harms, and establish new baselines for harm mitigation and multilingual alignment. our key findings and contributions can be enumerated as follows: ## experimental setup we use the 8b version of the aya 23 model @xcite as our pre-trained base model. we choose this model because it has been established as state-of-the-art in its size class, outperforming other widely used massively multilingual models such as aya-101 @xcite and @xcite . the experiments in this paper center around 6 specific languages: english, french, spanish, hindi, russian and arabic. this choice is guided by the intersection between the aya red-teaming dataset and the language capabilities of the aya 23 model. details of the training setup are available in appendix c. ## related work red-teaming large language models. in llm research, the term "red-teaming" has been used to describe systematic evaluations or attacks on llms to identify their potential risks and safety issues @xcite @xcite @xcite . initial efforts in this area focused on finding specific harmful inputs that could trigger dangerous outputs from the models @xcite . more recent studies have explored more structured jailbreaking attacks to discover adversarial prompts that can consistently bypass the safety measures of aligned llms @xcite @xcite . @xcite @xcite @xcite employ reinforcement learning from human feedback to model human preferences, while bianchi et al. (2023); qi et al. (2023); zhou et al. (2023) fine-tune llms using carefully designed benign data, aiming to align llms' behavior with human values. however, these studies focus on english-only models while our focus is on multilingual alignment techniques across diverse languages and local contexts, aiming to reveal cross-lingual transfer. harmful content in multilingual settings. ensuring safety and mitigating harm in multilingual contexts has become a critical focus in llm research. the xsafety benchmark (wang et al., 2023b) is the first multilingual safety benchmark for llms, covering 14 safety issues across 10 languages, which reveals that llms produce more unsafe responses for non-english queries, highlighting the need for better multilingual safety alignment. deng et al. (2024b) and li et al. (2024a) identify multilingual jailbreak challenges, proposing datasets and mitigation methods that significantly enhance model defense. shen et al. (2024) examine safety challenges across different languages, finding that llms generate more unsafe and irrelevant responses to malicious prompts in lowerresource languages. additionally, pozzobon et al. (2024) explore the complexities of multilingual toxicity mitigation, comparing finetuning and retrievalaugmented techniques across 9 languages, and providing insights into the effects of translation quality and cross-lingual transfer. both üstün et al. (2024) and aryabumi et al. (2024) conduct extensive multilingual safety evaluations -üstün et al. ( 2024 ) propose a novel multilingual safety context distillation approach which uses synthetic data to inherit refusal guardrails from a performant model. they distill safety preambles into the model for safetyrelevant contexts and show reductions in harmful generations for adversarial prompts by 78-89% as judged by human experts. in this work, we address critical gaps by creating a human-annotated dataset in 8 languages, covering a wide variety of harm categories, to reduce harmfulness without compromising helpfulness. culturally-sensitive scenarios in nlp. recent studies emphasize the importance of cultural variation in nlp models @xcite . @xcite found that fine-tuning english-only models on survey data improves cross-country in-ference but reduces accuracy for english norms. @xcite introduced probes to examine cross-cultural values in pre-trained models, revealing weak alignment with established values surveys. @xcite scaled the word embedding association test to 24 languages, uncovering significant social biases. @xcite demonstrated that llms encode differing moral biases in different languages. llms have been shown to also have a low agreement with human judges when detecting culturally-specific toxic language @xcite . efforts to address objectionable content generation include new adversarial attack methods with high transferability @xcite and a framework considering linguistic and cultural differences in nlp systems @xcite . cross-cultural studies on moral dilemmas show universal patterns and country-level variations influenced by relational mobility @xcite . @xcite found cross-lingual vulnerability in ai safety mechanisms highlighting a need for robust multilingual safeguards. our research focuses on developing multilingual guardrails across diverse languages and cultural contexts. ## conclusion in this work, we demonstrate the importance of a wide variety of alignment methods to tackle the dual challenge of mitigating both local and global harms across an array of diverse languages. this bifurcation becomes especially important in uncovering the subtleties and contrasts that are attached to each language and thus, at a larger level, to each associated culture. we demonstrate the effectiveness of several algorithms, from simple supervised fine-tuning to offline preference optimization, with dpo(sft) emerging as the method that shows a well-balanced trade-off between reducing the overall harmfulness of a model and impressive general performance. more interestingly, we observe consistent trends across all languages, showing that one language does not have to be optimized at the expense of another. furthermore, our methods show steady improvements in safety performance across both local and global harms regardless of the type observed during training. we hope that this work draws the attention of the research community to the very crucial subject of multilingual safety alignment and the release of the aya red-teaming dataset acts as a catalyst for the same.