GAMBIT +: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics (2025)

Abstract: Gender bias in machine translation (MT) systems has been extensively documented, but bias in automatic quality estimation (QE) metrics remains comparatively underexplored. Existing studies suggest that QE metrics can also exhibit gender bias, yet most analyses are limited by small datasets, narrow occupational coverage, and restricted language variety. To address this gap, we introduce a large-scale challenge set specifically designed to probe the behavior of QE metrics when evaluating translations containing gender-ambiguous occupational terms. Building on the GAMBIT corpus of English texts with gender-ambiguous occupations, we extend coverage to three source languages that are genderless or natural-gendered, and eleven target languages with grammatical gender, resulting in 33 source–target language pairs. Each source text is paired with two target versions differing only in the grammatical gender of the occupational term(s) (masculine vs. feminine), with all dependent grammatical elements adjusted accordingly. An unbiased QE metric should assign equal or near-equal scores to both versions. The dataset’s scale, breadth, and fully parallel design, where the same set of texts is aligned across all languages, enables fine-grained bias analysis by occupation and systematic comparisons across languages.

## introduction while gender bias in machine translation systems is widely acknowledged and has been widely documented, the biases in the translations of genderambiguous terms, such as occupational titles, is a topic relatively unexplored, which has lately gained some traction @xcite . these biases are often reflected in the disproportionate assignment of a gender (e.g. masculine forms) to occupations when the gender of the subject is unknown. while overall much research has focused on bias in mt outputs, comparatively little attention has been * corresponding author: geofila@ails.ece.ntua.gr. for each source text, we provide two target versions differing only in the grammatical gender of the occupational term (masculine vs. feminine). all necessary adjustments for grammatical correctness, such as gendered adjectives, are applied consistently across both versions. an unbiased qe metric should assign equal or near-equal scores to the two versions, as there is no indication of gender in the source text. unlike prior work, which often aggregates bias analysis across all occupations or texts in general, gambit+ explicitly tracks the occupational terms mentioned in each text and links them to isco-08 codes (hereafter, isco codes) 3 . this enables fine-grained, occupation-level analyses and facilitates the study of stereotypical patterns, as previous studies have shown that certain occupations are more likely to be translated in a gendered manner according to societal stereotypes @xcite . the parallel design of the dataset ensures that all source texts are aligned across target languages and both gendered versions, enabling consistent and controlled benchmarking of qe metrics. with this resource, researchers can investigate not only overall tendencies of qe systems but also nuanced, occupation-specific, and cross-lingual patterns, providing a more comprehensive understanding of how gender bias manifests in translation evaluation. as part of the challenge set subtask of the shared task on automated translation quality evaluation systems at wmt 2025 @xcite , gam-bit+ was used to evaluate a set of established qe metrics as well as participant submissions to the shared task. this allowed us to benchmark the performance of different metrics in a controlled, fine-grained setting and to assess how gender bias manifests in realworld qe systems. in this paper, we present and discuss the results of these evaluations, highlighting both systematic tendencies and occupation-specific patterns, and providing insights into the current limitations and strengths of existing qe approaches with respect to gender fairness. ## related work gender bias in machine translation has been widely documented, revealing persistent disparities influenced by societal norms, model design choices, and deployment contexts @xcite @xcite @xcite . numerous studies have examined the prevalence and consequences of such biases across languages, cultural settings, and mt architectures @xcite @xcite @xcite , underscoring the need for robust evaluation and mitigation. a particularly relevant line of work focuses on occupational bias in mt, where stereotypes associated with specific professions affect translation choices classifying occupations endorsed by the international labour organisation (ilo). it provides a hierarchical structure that categorizes jobs into four levels of increasing granularity, using a digit-based coding system. more details: https://ilostat.ilo.org/methods/ concepts-and-definitions/classification-occupation/ @xcite @xcite . related efforts in other nlp tasks have explored gender ambiguity in question answering @xcite and coreference resolution @xcite @xcite . in mt, proposed strategies for handling ambiguity include generating all grammatically correct gendered translations @xcite , and disambiguating inputs before translation @xcite . resources supporting these investigations range from knowledge graphs @xcite to multilingual benchmarks and challenge sets @xcite . mitigation strategies have included model fine-tuning, data balancing, and adaptive learning @xcite ; costa-jussà and de jorge, 2020), as well as gender-neutral translation approaches @xcite and benchmarks for evaluating them @xcite @xcite . a central element in all these efforts is evaluation, since quality estimation metrics determine what counts as a "good" translation and thus influence mt system development. while several studies have examined whether evaluation metrics for natural language generation exhibit social biases-such as @xcite , who compared n-gram-and model-based metrics, @xcite , who quantified different bias types, and @xcite , who measured race and gender stereotypes-very few works have explored gender bias in mt qe metrics. one exception is @xcite , which conducted a multifaceted analysis of qe methods and demonstrated that the metrics themselves can be biased, potentially perpetuating the very stereotypes they are used to assess. a key limitation of existing work on gender bias in qe is the lack of suitable datasets, particularly for studying gender ambiguity. many challenge sets, such as winomt @xcite and mt-geneval @xcite , focus on cases where gender can be resolved via coreference or other contextual cues. the must-she corpus @xcite similarly contains audio and textual cues that reveal gender (e.g., speaker's voice, pronouns, named entities). conversely, datasets containing true gender ambiguity @xcite are often limited in language coverage, consist of isolated sentences, and lack occupational diversity, restricting fine-grained analysis. in contrast, our challenge set builds on @xcite , where occupational mentions are inherently genderambiguous and no gold-standard gendered translation exists. this enables us to test whether qe metrics assign different scores to two theoretically equivalent translations differing only in the grammatical gender of the occupation, offering a controlled setting for investigating metric bias in ambiguous contexts. ## challenge set creation the creation of the challenge set, gambit+, was grounded in the gambit dataset @xcite , which covers all isco occupations (all 4-digit codes) and contains the corresponding isco codes and names, with english texts distributed evenly across five formats: short stories, brief news reports, short statements, short conversations, and short presentations, in which the given occupation appears. in these texts, the gender of the occupation is not explicitly indicated. for example, in the sentence "the professor delivered an engaging lecture on generation modification by the auditorium", shown in in gambit+, these gender-neutral english sentences were translated into gendered target languages, producing parallel masculine and feminine versions of each sentence while preserving all other aspects of the translation. in addition, we extended the source language coverage to include genderless languages such as turkish and finnish, alongside english, ensuring a diverse and balanced dataset. in total, the challenge set comprises 29,415 source instances (9,805 each of the three source languages), which were translated into 11 target languages, resulting in 323,565 triplets, each consisting of the source sentence with a masculine and a feminine translation. ## evaluation setup & analysis approach for the evaluation, we prepared a dataset in which each entry consisted of a source text (s) in one of three source languages (english, turkish, finnish) and, for each target language, two corresponding hypotheses: one where the occupation appeared in the translated masculine form (t male ) and another where it appeared in the translated feminine form (t f emale ). the two hypotheses were constructed to differ solely in the grammatical gender marking of the occupation, with all other aspects of the sentence kept identical. this dataset was then provided by the shared task organizers to the participating teams, who ran their metrics and returned their scores for each hypothesis in every source-target language pair, while the organizers themselves used a set of baseline metrics on our dataset too. these returned results form the basis of the subsequent analysis. the source languages were 3 (english, turkish, finnish) and the target languages were 11 (arabic, czech, greek, spanish, french, icelandic, italian, portuguese, russian, serbian, ukrainian), leading to a total of 33 distinct source-target language pairs. the organizers returned results for 11 evaluation metrics, 3 baseline metrics (sentinel-cand @xcite , sentinel-src @xcite , and cometkiwi22 @xcite ), and 8 submissions (uva-mt @xcite , ranked-comet @xcite , metricx-25-qe @xcite , metricx-25 @xcite , basecomet @xcite , polycand-1 @xcite , polycand-2 @xcite , and polyic-3 @xcite ). all metrics, except for uva-mt, were run on the complete set of 33 source-target language pairs. the uva-mt metric was evaluated only with english as the source language, covering all corresponding target languages. sentinel-src relies exclusively on the input, which results in identical scores being assigned to both gendered forms; therefore, it is excluded from our subsequent analyses 5 results and discussion ## gender density analysis different human-evaluation approaches of translation penalize errors (such as using the wrong gender) differently: direct assessment applies a conceptual approach, penalizing each error only once @xcite , whereas mqm @xcite penalizes every occurrence of the error in a text. to better understand how qe metrics capture gender related aspects, we analyse the correlation between the as presented in ## conclusions we presented gambit+, a large-scale, fully parallel challenge set containing paired masculine and feminine translations of gender-ambiguous occupations across 33 source-target language pairs, and used it to benchmark 10 qe metrics. our evaluation revealed consistent, statistically significant score shifts driven solely by grammatical gender-often more pronounced in certain language pairs and occupations-with a clear overall tendency to favor mascu-line forms. these findings indicate that current qe metrics are not gender-fair and should be systematically audited and calibrated. future work will broaden the scope of our analysis to include a wider range of qe metrics, with a focus on identifying specific characteristics that make them more susceptible to gender bias. we also plan to investigate the interplay between mt systems and qe metrics, exploring how system outputs and metric evaluations align or diverge in terms of gender bias. we also plan to extend gambit+ with translations of varying quality to investigate whether translation quality influences the biases of qe metrics-for example, whether poor translations make gender distinctions more or less apparent due to incorrect gender agreements across the sentence. finally, we aim to extend the use of gambit+ beyond mt, testing its applicability and value in evaluating gender bias in other natural language generation tasks.