John vs. Ahmed: Debate-Induced Bias in Multilingual LLM s (2024)

Abstract: Large language models (LLMs) play a crucial role in a wide range of real world applications. However, concerns about their safety and ethical implications are growing. While research on LLM safety is expanding, there is a noticeable gap in evaluating safety across multiple languages, especially in Arabic and Russian. We address this gap by exploring biases in LLMs across different languages and contexts, focusing on GPT-3.5 and Gemini. Through carefully designed argument-based prompts and scenarios in Arabic, English, and Russian, we examine biases in cultural, political, racial, religious, and gender domains. Our findings reveal biases in these domains. In particular, our investigation uncovers subtle biases where each model tends to present winners as those speaking the primary language the model is prompted with. Our study contributes to ongoing efforts to ensure justice and equality in LLM development and emphasizes the importance of further research towards responsible progress in this field.

## introduction language models have become an essential tool for various applications, from nlp to human-computer interaction. as these models continue to advance, the focus on their safety and ethical implications has become increasingly important. this is especially true for models that cater to specific languages or regions, where cultural nuances and contexts play a significant role in ensuring safe and responsible use @xcite . large language models (llms), such as gpt, are initially pretrained on vast amounts of websourced text. this text can contain biases and toxic content, leading to llms that may perpetuate these issues when fine-tuned for specific tasks @xcite . @xcite investigates biases in large language models (llms) in healthcare contexts. using simulations with openai's gpt-4, physicians make life-and-death decisions in resourcescarce scenarios. results show consistent biases based on demographic traits like race, gender, age, political affiliation, and sexual orientation. physicians tend to favor patients similar to themselves. for instance, they often save patients with their own demographic characteristics and political affiliation. the study highlights the significant biases in publicly available chatbot llms, raising concerns about their potential negative impact on patient outcomes if used in clinical care without precautions. to mitigate bias and toxicity, methods like direct preference optimization (dpo) @xcite and reinforcement learning from human feedback (rlhf) @xcite have been proposed. these methods align generative models with user intent after the fine-tuning phase, aiming to make the models more helpful, truthful, and harmless. while dpo and rlhf have been deemed effective and are widely-used, there is evidence suggesting that they may only obscure deeply-rooted biases, making such biases less apparent but not necessarily eliminated @xcite . @xcite analyze the safety of arabic and english language models and find that arabic models had higher rates of generating harmful and biased content, particularly in the domains of religion and politics. @xcite empirically study the multilingual safety of four widely used llms. the results show that all llms produce significantly more unsafe responses for non-english queries than english ones, indicating the necessity of developing safety alignment for non-english languages. these studies underscore the necessity for additional research on the safety of language models, particularly in handling sensitive topics based on prompt language. furthermore, they emphasize the importance of developing safeguards and implementing best practices to ensure the responsible use of these models. most existing research on bias in language models is mainly focused on english, leading to a significant research gap concerning other widely spoken languages, such as arabic and russian. consequently, there is a substantial lack of comprehensive studies that explore and understand biases in these languages. additionally, there is an unexplored area in the methodology of bias research, specifically the use of debate-induced methods to uncover biases in language models. these methods, which involve using argumentative text to provoke and reveal biases, may offer novel insights into how biases are embedded and can be mitigated. this research aims to address these two primary limitations by extending the focus beyond english and incorporating debate-induced methods to provide a deeper understanding of biases in language models. in this study, we aim to explore language model safety. we focus on evaluating biases across various languages and contexts in gpt-3.5 foot_0 and gemini @xcite . in particular, we propose and utilize a novel debate-based prompting setup that we claim is effective in exposing the deeply-rooted bias in llms. we employ a multilingual approach involving arabic, english, and russian. through our designed prompts and scenarios, we attempt to evaluate cultural, political, religious, racial and gender biases in such large models and contribute to the ongoing discourse on fairness and equity in nlp. the rest of the paper is organized as follows: section 2 is about related work. we introduce our prompt template that is used to generate the debates in section 3. in section 4, we present our methods involving the models we use and how we evaluate them. our experiments are in section 5 and section 6 is a walk-through and discussion of our findings. we conclude in section 7, list our limitations in section 8, and discuss ethical considerations in section 9. we now review related work. ## related work there have been several recent efforts to examine the safety of language models. @xcite investigates the presence and impact of gender biases in large language models (llms). they present a novel testing paradigm that differs from previous methods, using sentences designed to reveal biases in models' occupation-related assumptions based on gendered pronouns. their findings indicate that llms are significantly more likely to align occupations with stereotypical gender roles, reflecting societal biases. moreover, the paper discusses how llms often fail to recognize the inherent ambiguity in pronoun references in sentences, leading to biased and factually inaccurate explanations for their choices. furthermore, @xcite proved explicit and implicit gender bias using a conditional text generation mechanism, concluding that a larger model doesn't necessarily equate to greater fairness. @xcite investigates in large pre-trained language models gender bias towards politicians, which varies significantly among the examined languages. arora et al. (2023) utilized hofstede's cultural dimensions theory and questions from the world values survey to investigate how pre-trained language models (plms) reflect cultural differences in values. their study involved converting survey questions into cloze-style probes and running them through three multilingual language models (mbert, xlm, and xlm-r). they found that the values embedded in these popular multilingual models varied across cultures but only weakly aligned with established values surveys. this discrepancy underscores the complexities and challenges in ensuring plms accurately reflect diverse cultural values. @xcite delve into whether english language models possess knowledge concerning moral norms in various countries. its findings anticipate that empirical moral norms across countries are less favourably compared to previously reported english moral norms. moreover, in "having beer after prayer? measuring cultural bias in large language models" @xcite , the authors address the critical issue of cultural bias in multilingual and arabic monolingual large language models (llms). they unveil the models' inclination towards western culture, demonstrating this bias through a series of experiments involving story generation, named entity recognition (ner), sentiment analysis, and text infilling tasks. utilizing camel @xcite , a novel resource consisting of 628 prompts and 20,368 entities spanning arab and western cultures, the study reveals llms' tendency to stereotype and misrepresent arab culture, often favoring western-centric content. despite focusing on arabic contexts, the findings highlight a broader chal-lenge in ensuring llms accurately reflect diverse cultural nuances. similarly, in @xcite the authors introduce eticor, an etiquettes corpus comprising texts about social norms from five different global regions. this corpus aims to evaluate large language models' (llms) knowledge and understanding of region-specific etiquettes, highlighting the cultural sensitivity necessary for dayto-day human interactions. initial experiments with state-of-the-art llms reveal their limitations in accurately representing non-western etiquette, underscoring the models' inherent biases towards western norms. @xcite employ a novel empirical design to assess political bias within chatgpt. they analyze chatgpt's responses to a set of questions mimicking the political compass questionnaire, intending to impersonate political inclinations toward specific parties in the united states, brazil, and the united kingdom. the study reveals a significant systematic bias of chatgpt towards left-leaning political parties: the democrats in the us, labor in the uk, and lula's supporters in brazil. this tendency raises concerns about the potential impact of such biases on political and electoral processes. moreover, @xcite introduced two metrics to measure political bias in gpt-2-generated text: indirect bias and direct bias. indirect bias assesses the political bias of texts generated using neutral prompts by comparing the bias in texts generated with specific keywords to those generated with all keywords in an attribute. direct bias evaluates the difference in bias when texts are generated using prompts with explicit political triggers (e.g., "democrat" or "republican"). their findings revealed a predominant liberal-leaning bias in gpt-2 outputs, influenced by the context's sensitive attributes. then it introduces a novel reinforcement learning framework aimed at mitigating such biases without the need to retrain the model or access the original training data. this framework, adaptable to both word-embedding and classifier-guided approaches, effectively reduces political bias across various attributes while maintaining the readability and semantic coherence of the generated text. @xcite undertake a rigorous examination of trustworthiness in gpt-3.5 and gpt-4 models on multiple dimensions, including toxicity, stereotype bias, adversarial robustness, out-ofdistribution robustness, privacy, machine ethics, and fairness. this work presents a novel framework for evaluating the complex trustworthiness of large language models (llms), revealing significant vulnerabilities, such as the ease of eliciting toxic and biased output, privacy breaches, and susceptibility to adversarial prompts. the study notably finds that while gpt-4 often outperforms gpt-3.5 in standard benchmarks, it is more prone to manipulation through carefully crafted prompts, underscoring the nuanced trade-offs between model capabilities and vulnerabilities. the comprehensive nature of this assessment provides a foundation for efforts to improve the safety, fairness, and reliability of llms. @xcite introduce a comprehensive framework to evaluate social biases in openended language generation. the paper presents the bias in open-ended language generation dataset (bold), a large-scale dataset consisting of 23,679 english text generation prompts across five domains: profession, gender, race, religion, and political ideology. alongside this dataset, the study proposes new automated metrics for assessing social biases through dimensions of toxicity, psycholinguistic norms, and text gender polarity. evaluations conducted on texts generated by three popular language models reveal pervasive social biases, with generated content often reflecting stereotypical and prejudiced views, particularly against historically disadvantaged groups. in a recent paper, alghamdi et al. (2024) address the critical need for assessing the trustworthiness of large language models (llms) in arabic. introducing aratrust, a benchmark for evaluating the trustworthiness of arabic llms, the study spans multiple dimensions including ethics, safety, privacy, and offensive language. aratrust consists of 516 human-crafted multiple-choice questions covering a wide range of topics relevant to arabic cultural and societal norms. evaluations of several llms, including gpt-4, acegpt 7b and jais 13b, reveal significant disparities in performance, particularly highlighting the superior trustworthiness of proprietary models over open-source alternatives. this work sheds light on the under-explored area of llm safety and trustworthiness in non-western languages, marking a step towards the development of safer and more culturally aligned ai systems for arabic-speaking users. ## data we introduce a novel prompt template that we use to generate model outputs over multiple trials. we then use the generated output to quantify and evaluate model bias. as seen in figure 5 , our prompt template revolves around generating a debate between two sides representing two opposing viewpoints. this debate-based setup is an indirect way to uncover the deeply rooted bias in the model, bypassing the safety alignment in a similar fashion to @xcite . constraining the debate outcome to be binary makes quantifying the bias straight-forward. the prompt template is filled in using some of the domain and debate outcome examples that are presented below in section 4.1. across all @xmath0, this prompt is provided in three deliberately selected distinct languages for this task: english, arabic, and russian. each language offers unique linguistic nuances and cultural perspectives, enriching the diversity of responses generated. by presenting the prompt in multiple languages, we aim to explore how cultural backgrounds and linguistic contexts influence the portrayal and interpretation of the debate. this multilingual approach ensures a comprehensive examination of the topic across different domains and debate outcomes. ## experiments we prompted two models with 36 carefully selected questions across five domains in three different languages. our investigation encompasses a singleturn probing methodology. in single-turn probing, both the cultural context and the probe are provided in a single prompt, facilitating direct assessment. each prompt is generated 20 times and the win rate is calculated. ## conclusion drawing on our crafted prompts for story generation, we leveraged debates as a probing mechanism to unveil biases inherent in gpt-3.5 and gemini. across cultural, political, racial, religious, and gender domains, we meticulously scrutinized the coherence of responses from these conversational agents. our comprehensive analysis uncovered a discernible inclination within both chatgpt and gemini, particularly evident in their determination of debate outcomes related to specific cultures, religions, or political affiliations. interestingly, these biases were not readily apparent without the structured framework of debate. notably, a significant bias was observed when prompting in arabic, regardless of the topic or domain. conversely, russian prompts yielded the most neutral outcomes when compared to english and arabic in both gpt-3.5 and gemini. these findings underscore the persistent biases in models despite efforts to align and eliminate them, highlighting the imperative to enhance model safety precautions. as a direction for future work, further investigation into refining and augmenting these safety measures is warranted.