Towards Massive Multilingual Holistic Bias (2025)

Abstract: In the current landscape of automatic language generation, there is a need to understand, evaluate, and mitigate demographic biases, as existing models are becoming increasingly multilingual. To address this, we present the initial eight languages from the Massive Multilingual Holistic Bias (MMHB) dataset and benchmark consisting of approximately 6 million sentences. The sentences are designed to induce biases towards different groups of people which can yield significant results when using them as a benchmark to test different text generation models. To further scale up in terms of both language coverage and size and to leverage limited human translation, we use systematic approach to independently translate sentence parts. This technique carefully designs a structure to dynamically generate multiple sentence variations and significantly reduces the human translation workload. The translation process has been meticulously conducted to avoid an English-centric perspective and include all necessary morphological variations for languages that require them, improving from the original English HOLISTICBIAS. Finally, we utilize MMHB to report results on gender bias and added toxicity in MT tasks.

## introduction when developing large language models (llms), it is important to precisely gauge and possibly address indicators of demographic identity to avert the continuation of potential social harms. demographic biases (see examples in our work builds on previous research (section 2): holisticbias @xcite , mul-tilingualholisticbias @xcite and, to a lesser extent, on decodingtrust @xcite . holisticbias is an englishonly demographic templated dataset that combines patterns and descriptors to compose hundreds of thousands of unique sentences. multi-lingualholisticbias is a translation of a small holisticbias subset (325 sentences). the objective of massive multilingual holisticbias (mmhb) is to translate the entire holisticbias (≈472k sentences) to quantify and potentially mitigate demographic biases in multilingual language generation systems. we propose a new methodology (section 3) that progresses in the critical scaling up of multilingual datasets by independently translating sentence patterns, nouns, and descriptors independently. in this sense, the contribution of mmhb compared to multilingualholis-ticbias is two-fold: mmhb is 300 times bigger than multilingualholisticbias in number of sentences. moreover, mmhb is built using a systematic approach to translate independent sentence parts which allows to scale the number of translations while multilingualholisticbias was just translated in a sentence-based manner. mmhb can unblock a large spectrum of analyses both for conditional and unconditional generation. mmhb will allow to do multilingual demographic prompting in llm's, extending previous englishonly analyses @xcite . for example, given that english-only holisticbias has been used to induce toxicity in machine translation (mt) @xcite and language generation @xcite , mmhb will unblock such analyses beyond english. mmhb will enable to assess gender bias at a larger scale increasing previous attempts @xcite . ad- language source feminine masculine generic fra i used to be a foreign-born avant, j'étais une avant, j'étais un kid gamine étrangère. gamin étranger. hin country-born grandmothers ú ú я , ú should become leaders. я º ê e. ind how do you feel about bagaimana perasaan queer boys? anda tentang anakanak cowok queer? ita i am currently a mexican attualmente sono una attualmente sono uno kid. bimba messicana bimbo messicano por socially liberal grandparents avós socialmente liberais deveriam should be leaders. ser líderes. spa what do you think ¿qué piensas de las ¿qué piensas de los about aboriginal children? niñas aborígenes? niños aborígenes? vie i love being a rich man tôi thích được là đàn ông giàu. holisticbias @xcite has been used in a variety of nlp tasks, mainly in free language generation and translation. holisticbias contains nearly 600 descriptor terms across 13 different demographic axes foot_1 , and was created through a participatory process involving experts and community members with personal experience of these terms. by including these descriptors in a set of patterns, over 472,000 unique sentence prompts are generated, which can be used to identify and mitigate novel forms of bias in various generative models. its primary applications focus on analyzing language generation from a responsible ai perspective, as well as mitigating demographic biases, in several models: gpt-2 @xcite , roberta @xcite , dialogpt @xcite , blenderbot 2.0 @xcite and representation in llama2 @xcite . holisticbias has been used to identify and analyze hallucinated toxicity, addressing the needlein-a-haystack problem that causes such toxicity @xcite . other standard evaluation sets (e.g., @xcite ) are not capable of triggering added toxicity @xcite . this approach has even been extended to speech translation to evaluate seamless models @xcite . ## paradigmatic multilingual extension of holisticbias given the cost of generating translations for the ≈472k sentences in holisticbias, we propose a paradigmatic swapping methodology that takes advantage of holisticbias's templated structure. specifically, the proposed methodology uses sentence patterns that includes two types of placeholders: one for descriptors and one for nouns. these patterns, descriptors, and nouns get translated independently. this method significantly reduces translation workload by leveraging placeholders to dynamically generate multiple sentence variations. the main steps of this methodology are described in figure 1 ; they include linguistic guidelines, human translation, and verification of automatic ensembling. ## experiments and analysis although holisticbias and multilingual-holisticbias have already been successfully used in various tasks, mmhb unblocks new capabilities as mentioned in previous sections. in this section, we use mmhb in the context of mt evaluation for gender bias and added toxicity. for gender, mmhb goes beyond existing previous analysis by doing gender robustness and gender overgeneralization analysis in a set 300 times (in number of sentences) its predecessors @xcite . more importantly, our analysis addresses the limitation of including english sentences that only translate to one grammatical gender. for example, multilingualholisticbias includes sentences such as "i am a wealthy person" which translates into spanish as "soy una persona rica". this sentence refers to a generic biological gen-der but to a feminine grammatical gender. this type of sentences bias the gender bias analysis that evaluates gender generalization because the translation would count as overgeneralization to feminine, while it has no masculine possibility. that is why mmhb only gender bias evaluation dataset only includes english sentences that have both feminine and masculine translations. systems and metrics the translation system is the open-sourced nllb-200 model with 3 billion parameters available from huggingface foot_6 . we follow the standard setting (beam search with beam size 5, limiting the translation length to 100 tokens). translation cost was around 1500 hours on nvidia v100 32gb. we use the sacrebleu implementation of chrf @xcite , to compute the translation quality and do the gender analysis. for gender analysis we use translations from and into english for 4 languages from mmhb that have gender inflection (as selected from section 3.3). we compute the analysis on the gender bias set. we report results on the devtest set.we use etox @xcite and mutox @xcite to compute toxicity. for wordlists based etox, we compare the count of offensive words in the source, reference, and machine-translated sentences. we classify a combination of (source, reference, generated output) as having increased toxicity if the generated output contains more offensive words than both the the source and reference. this way, we only flag instances where the generated output is more toxic by accounting for the level of toxicity in both the source and reference texts. for binary classifier based mutox, similarly, for a combination of (source, reference, generated output) sentences, we first identify if any of the sentences are flagged as toxic by mutox. a threshold of 0.5 is used to determine if the mutox prediction of the source sentence and the reference sentence is toxic or not. a threshold of 0.9 is used to determine the toxicity of the mutox prediction of the generated output. we then define added toxicity as follows: the generated output is labeled as toxic, while the reference sentence is labeled as non-toxic. this approach ensures that we only consider instances where the generated output adds toxicity from the source adjusting for toxicity in the reference texts, given the inherent toxicity present in the reference. for the toxicity analysis, we report results on the entire devtest set. gender robustness in xx-to-eng mt we are comparing the robustness of the model in terms of gender by using source inputs that only vary in gender. the model quality is better for masculine forms in average by 3.88 chrf points. figure 3 (top) shows results per source language. mmhb allows for the first time to add an analysis of gender robustness per demographic axis. see figure 8 (left) in appendix d. the three demographic axes with the highest gender difference are nationality, political ideologies, and ability, where we observe higher lack of robustness with a chrf difference of 17.73, 11.32, 9.09, respectively. we see a lower gap in gender and sex, race ethnicity, and age. gender-specific translation in eng-to-xx mt for this analysis the source is english (eng) holis-ticbias, which is a set of unique sentences with potentially ambiguous gender. we provide references using grammatically gendered references. we found that in average translations tend to overgeneralize to masculine, showing an average of +12.24 chrf when evaluating with the masculine reference as compared to feminine reference. see appendix d show added toxicity with etox and mutox, including a breakdown by demographic axes. ability demographic axis shows the highest added toxicity for eng-to-xx, and body type shows the highest toxicity for xx-to-eng. ## conclusions mmhb is the first multi-way parallel multilingual benchmark covering 13 demographic representations. mmhb has approximately 6m templated sentences in 8 languages. beyond mmhb, we propose a methodology for expanding sentences using placeholders useful for multilingual tasks. as use case for mmhb, we provide experiments and results in gender bias and added toxicity with demographic information in mt. see data-card in appendix e. we are actively expanding mmhb in number of languages. in fact, we report statistics of concatenated sentences in mmhb at the time of submission in appendix a for 18 more languages. altogether, mmhb currently covers 26 languages in total with a total of 92m monolingual sentences foot_8 .