GENDER 1 PERSON : Test Suite for Estimating Gender Bias of First-person Singular Forms (2025)

Abstract: The gender1person test suite is designed to measure gender bias in translating singular first-person forms from English into two Slavic languages, Russian and Serbian. The test suite consists of 1,000 Amazon product reviews, uniformly distributed over 10 different product categories. Bias is measured through a gender score ranging from -100 (all reviews are feminine) to 100 (all reviews are masculine). The test suite shows that the majority of the systems participating in the WMT-2025 task for these two target languages prefer the masculine writer’s gender. There is no single system which is biased towards the feminine variant. Furthermore, for each language pair, there are seven systems that are considered balanced, having the gender scores between -10 and 10.Finally, the analysis of different products showed that the choice of the writer’s gender depends to a large extent on the product. Moreover, it is demonstrated that even the systems with overall balanced scores are actually biased, but in different ways for different product categories.

## introduction while english does not have many morphological forms related to gender, the two target languages do so. in those languages, gender marking exists not only for pronouns and animate nouns, but also for nouns, adjectives, verbs, determiners and numbers. if the text is written in the first-person singular form and no information about the gender of the author is provided, the translator can choose any of the two binary foot_0 genders. one of the most frequent affected pos tags are adjectives and past or passive participles. for example, "i am happy that i bought this" can be translated into serbian in two ways, "srećan/srećna sam što sam ovo kupio/kupila", depending on the writer's natural gender. this may result in translation errors, mismatches and inconsistencies, as well as in gender bias. our test suite is designed to measure bias of this type of gender in translations from english into russian and serbian. it consists of a carefully selected set of user reviews about amazon products, because these texts are written in the first-person form and therefore very convenient. the test suite also enables the analysis of writer's gender depending on the product category. although currently covering two target languages, it can easily be extended to more languages with similar rules for first-person singular gender. our main motivation was the results of our experiments reported in @xcite . we found some interesting tendencies regarding the writer's gender in user reviews of amazon products from the dihutra corpus @xcite . however, this corpus is designed for investigating differences between human and machine translations, but it is not tailored for exploring gender. the corpus is relatively small, only 196 reviews in total, and one third of them do not contain any indicator of the writer's gender. therefore, the reported results (especially for different products), while interesting, were not fully reliable. the test suite presented in this paper, is much bigger and contains 1000 reviews. ## related work while there is a large portion of work dealing with different types of gender bias, there are not many studies focussing on the first-person constructions. for example, @xcite propose automatic generation of both gender variants for the first person in arabic nmt translations. our test suite also enables analysing bias depen-dence on product category. similarly, bias variation was addressed in @xcite who reported that data sets for specific tasks (e.g. cooking) contain significant gender bias and, furthermore, models trained on these data sets further amplify existing bias. our test suite uses a gender score as a metric. cho et al. (2019) also proposes a measure of gender bias, however, in a completely different context: the metric measures the relation between gender and positive/negative expressions or occupations. there exist other test suites. for instance, @xcite design a test suite for evaluating gender bias in mt related to occupational nouns. their method was developed for eight target languages, including russian. vanmassenhove and monti (2021) create an english-italian test suite with a focus on the resolution of natural gender. the authors provide word-level gender tags on the english source side and multiple alternative gender translations, where needed, on the italian target side. @xcite present a test suite to investigate systems' ability to correctly translate the gender of the speaker int he context of occupations and professions (e.g. "i am a doctor"). the test suite from @xcite measures the gender resolution tendencies of mt systems in literary-style dialogues. however, to the best of our knowledge, none of the available test suites addresses the writer's gender from a general point of view. we believe that the presented test suite will add value to studies addressing gender in the area of machine translation and natural language processing. ## test suite creation the test suite consists of 1 000 user reviews of 10 different product categories extracted from the publicly available repository "amazon reviews 2023" foot_1 @xcite . the repository contains reviews written between 1996 and 2023 divided into more than 30 different product categories. for our test suite, we selected 10 categories, and extracted the first 100 reviews from each according to the following criteria: the threshold for the pronoun "i" is set to ensure that there will be enough instances of first-person singular gendered words in the translations. the length limit is set to avoid very long reviews, and very short reviews are automatically discarded due to the threshold for the pronoun "i". the statistics of the obtained test suite is shown in in most of the reviews, the gender of the writer is not specified by any information in the english source. explicit gender cues (e.g. "i'm a man/woman" or "male/female", "i was pregnant") can be found only in 1.5% of the reviews: there are 14 explicitly feminine reviews and one masculine. in addition, some of the reviews contain potential gender cues, namely "husband" and "wife" which used to be explicit and therefore can influence the choice of the writer's gender. in total, there are 17 reviews with "my husband" and 16 reviews with "my wife". all in all, there are notably more explicit feminine cues, while potential cues are balanced. therefore, it can be expected that a translation with balanced gender distribution might contain slightly more feminine reviews. the distribution of the gender cues over product categories can be seen in appendix a.1. ## evaluation method the evaluation method follows the principles from the manual gender labelling described in (popovic and lapshinova-koltunski, 2024), but is fully automatic. it consists of three steps: (1) word-level annotation (identifying gendered words related to first-person singular), (2) review-level annotation based on the word-level gender labels, and (3) calculation of gender score based on the review-level gender labels. ## summary and outlook summary the presented test suite is designed for analysis of the first-person singular gender (speaker's or writer's gender) in translations from english into russian and serbian. the gender score is defined to measure the balance between the two binary genders, masculine and feminine. the score ranges between -100 (fully feminine) and 100 (fully masculine), and values between -10 and 10 are considered as balanced. after using the test suite on wmt-2025 translation outputs to calculate language level scores, system level scores and product level scores, the main findings are: furthermore, three product categories are identified to be predominantly masculine and two as mostly feminine. automotive, musical instruments and video games are heavily biased towards masculine by all models (all scores are larger than 50). baby products and beauty and personal care are biased towards feminine, but to a lesser extent: scores are ranging from -80 to 99, and a few systems are balanced. overall, the results confirmed the similar tendencies reported in previous work for croatian and russian on a small scale (popovic and lapshinova-koltunski, 2024). given that one of the reported tendencies was that even human translators are generally inclined to opt more often for a masculine writer, and are also influenced by the product category, the wmt-2025 results from the test suite are not surprising. question is the nature of the translations with the label "x". it should be explored whether they are really gender-neutral, or an inclusive form was used, or the annotation script missed all words of interest, or there are possibly translation errors. as for mixed reviews, word-level gender scores could be added to include their contribution. another direction is analysis of reviews with gender cues (for instance specific words like pregnant, etc., and their possible separation into a sub-suite. an obvious direction is extending the test suite with more data. also, it can be easily extended to other languages with gendered first-person singular words, such as french, spanish, czech among others. also, other domains/genres apart from amazon product reviews should be considered. finally, a systematic experiment on prompt design should be carried out in order to use llms for word-level annotation and improve recall without deteriorating precision.