Gender-Fair Post-Editing: A Case Study Beyond the Binary (2023)

Abstract: Machine Translation (MT) models are well-known to suffer from gender bias, especially for gender beyond a binary conception. Due to the multiplicity of language-specific strategies for gender representation beyond the binary, debiasing MT is extremely challenging. As an alternative, we propose a case study on gender-fair post-editing. In this study, six professional translators each post-edited three English to German machine translations. For each translation, participants were instructed to use a different gender-fair language strategy, that is, gender-neutral rewording, gender-inclusive characters, and a neosystem. The focus of this study is not on translation quality but rather on the ease of integrating gender-fair language into the post-editing process. Findings from non-participant observation and interviews show clear differences in temporal and cognitive effort between participants and strategy as well as in the success of using gender-fair language.

## introduction gender bias in machine translation (mt) has been studied from different angles and a lot of work has been published for debiasing mt, however, only recently from the perspective of bias beyond a binary conception of gender. most approaches @xcite discuss it from a theoretical perspective. one exception is @xcite , who proposed a gender-tagging approach to translate from inflected to gender-neutral language with moderate success. gender-fair debiasing mt models is challenging, since there is a lack of datasets and even human translators find it difficult to select and correctly apply gender-fair language strategies. as a first step, we propose a case study to investigate gender-fair language within the context of postediting. post-editing approaches generally focus on speed, productivity, cognitive load, and quality @xcite . to the best of our knowledge, this is the first post-editing study to focus on gender-fair language. non-binary individuals have become increasingly visible, such as in tv series like one day at a time or sex education, and the number and type of strategies to linguistically represent them differs considerably across languages. gender-fair language (gfl) subsumes gender-neutral, that is, omitting any gender references, and gender-inclusive language, that is, linguistically including all genders. with grammatical differences and a multitude of strategies across languages, gender-fair translation is challenging for machines and humans. nevertheless, human and machine translators can act as ambassadors for gender equality beyond the binary by using gender-fair language. in this case study, we chose post-editing over translation from scratch to focus on the temporal and cognitive effort required to revise an existing translation in terms of gender references. six professional translators post-edited three machine translations from english to german, which contain references to non-binary individuals. while for english singular they has become predominant, in german there are many different strategies. participants were instructed to apply one specific approach per text, i.e., gender-neutral rewording, gender-inclusive characters, and neosystems. screen recordings allow to measure the temporal effort. post-experiment interviews provide insights into the cognitive load felt by participants depending on the strategy. finally, analyses of post-edited translations reveal the level of difficulty of the task. the less familiar a person is with gender-fair language, the more difficult it is to correctly detect and revise gender references. @xcite argue to only use gender-neutral strategies in mt and only utilise gender-inclusive forms where necessary not to lose important information. in our experiment, participants equally expressed a clear preference for a combination of these two strategies over neosystems. with results from this case study, we contribute to guidelines for integrating gender-fair language into the translation workflow. we show which gender-specific constituents are particularly challenging in the source and target text, which potentially provides inspirations to mt debiasing. ## related work research on gender-fair language in human and machine translation is still in its infancy and very few publications address the topic from either a translation studies or mt perspective @xcite . one exception is @xcite , who conducted a participatory workshop on both gender-fair human and machine translation, bringing together different stakeholders and working in a multidisciplinary team of experts from translation studies, mt, gender studies, and human-computer interactions. the results of this workshop highlight that strategy selection is highly dependent on different criteria, e.g., context, target audience, scope of the text, thus there being no "one-size-fits-all" solution @xcite . since, to the best of our knowledge, this is the first gender-fair post-editing study, we will introduce gender-fair translation from the perspective of translation studies and mt. @xcite and @xcite analysed, among others, the dubbed and the subtitled versions of the netflix series one day at a time in spanish and french. both found that translation strategies varied based not only on the version, i.e., subbed or subtitled, but also on the language variety, i.e., european and latin american spanish. in three of the four spanish versions, a non-binary character is addressed with female forms, and/or a literal translation of english singular they. only in the european spanish dubbing, the non-binary neopronoun "elle" is utilised. in the french subtitles for the series, the french indefinite pronoun "on" (one/we) is used. misiek (2020) analysed the polish translation of three different english language tv series that feature non-binary characters and found a complete omission of their gender identity. in croatian articles on sam smith's coming out as non-binary and movie translations @xcite , the masculine plural pronoun was found as a frequent strategy, which is an instance of misgendering. in a first attempt to debias nmt models beyond the binary, @xcite extended a gender-balanced corpus (saunders and byrne, 2020) by gender-neutral sentences with placeholders for gender inflections in german and french for training. for testing, they produced a genderneutral version of the winomt dataset @xcite and found a low overall accuracy and a tendency to over-generalise the use of exclusively gender-neutral language, even if the source text was clearly gendered. from a theoretical perspective, @xcite propagate gender-neutral strategies for machine translation and propose gender-neutral constraint-based algorithms at training time, wider contexts than sentence-level, and injecting external knowledge as possible approaches. furthermore, they highlight the difficulty of identifying gender references to be changed, e.g. the mother in motherboard might not be a candidate. ## preliminaries in order to provide a basis for gender-fair postediting, this section briefly introduces genderfair language strategies for english and german. as a notional gender language, english generally requires gender-specification in third-person singular pronouns (e.g. he/she/it) and in some specific nouns, usually in reference to kinship (e.g. mother/father) or professions (e.g. chairman/chairwoman) @xcite @xcite . to achieve gender-fair english, singular they and genderneutral nouns, e.g. chairperson, are often used to address non-binary people (apa style, 2019). other languages, such as german and italian, are grammatical gender languages and require extensive gender marking in pronouns, nouns and also in adjectives or participles @xcite . in german, for example, four different approaches can be identified, i.e., (i) gender-neutral rewording; (ii) gender-inclusive characters; (iii) gender-neutral characters and forms; and (iv) neosystems. in (i), sentences are structured in order to avoid gender-specification using, e.g. gender-neutral words such as person, indefinite pronouns, passive constructions and participial forms. in (ii), characters such as gender star (*) are used to separate male forms from female endings, e.g. leser*in (reader) to include all genders. in (iii), similar characters or new endings like "x" in lesx (reader) are used to question the gender binary. in (iv), a fourth gender in addition to masculine, feminine and neuter is introduced in the german language as in the case of lesernin (reader). several comprehensive overviews of gender-fair language in german are available @xcite @xcite . the universal acceptance of gender-fair language can and has been debated. for instance, vergoossen et al. (2020) provide four dimensions of resistance against the introduction of the genderfair pronoun hen in swedish, including distraction in communication, defending the status quo, and cisgenderism. however, linguistic change that does not come about naturally has always met initial resistance, but in the end facilitates social change towards gender equality @xcite . this, in turn, reduces linguistic and systematic identity invalidation and permits access to public spaces, e.g. restrooms, and services, e.g. personal identity cards. translators and machine translation can act as ambassadors for such change. ## method the proposed method, inspired by translation process research (tpr) @xcite and @xcite , combines non-participant observation, screen-recordings, retrospective interviews, and target text annotation. six professional translators with at least two years of practical experience were recruited. prior to the study, participants received instructions on the tasks, postediting guidelines by the translation automation user society (taus) foot_0 , and a handout on various strategies to gender-fair language in order to prepare for their participation. as shown in to ensure comparability of estimated pe times per text, readability scores were computed using the flesch-kincaid readability test @xcite . it takes into account the number of words and their length, but ignores semantics. the selected texts contained references both to nonbinary individuals as well as mixed-gender groups in english. german was selected as a target language being a grammatical gender language which needs extensive gender marking when compared to english. for the translation analysis, respectively 9, 12, and 10 gendered phrases were identified. these phrases were composed of different word classes, such as nouns, adjectives, articles and different types of pronouns, mostly singular they. the study was conducted online since it aimed for a most authentic and unintrusive experimental setting. translators could work in their familiar environment and were instructed to work under usual conditions. nevertheless, they were required to use one screen only in order for the whole postediting process to be recorded. during the process, a video conference was open in the background, on which the shared screen was recorded. subsequently, they were interviewed about their impressions, strategies, and which aspects of the study were particularly challenging. the interviews were conducted in german, transcribed ac2018 ) semantic transcription rules and then analysed by means of qualitative content analysis @xcite using the qualitative analysis software maxqda foot_2 . in order to analyse the gender-fair post-editing process, krings' (2001) division into temporal, technical, and cognitive effort was applied. the focus of this paper is on temporal and cognitive effort as well as an analysis of the final gender-fair translation and strategy. screen recordings were used to measure post-editing times and, thus, temporal effort. to test whether the different approaches to gfl had an impact on translation speed, a linear mixed effects model was run with packages imer4 and imertest in statistical analysis software r. gfl approach, participants' work experience and rates for gfl difficulty were used as independent variables while participants were used as a random factor. observation protocols were produced by means of non-participant observation and aimed at reconstructing the post-editing process. finally, interviews were used to gather data on the perceived cognitive effort of participants. in addition, gendered phrases in the post-edited texts were annotated based on the selected gender-fair language strategies and the success of their use. ## results after presenting the participant's profile, the temporal and subjective cognitive effort for each text and gender-fair language approach used are presented. furthermore, participants' impressions on the ease of using each strategy in post-editing are summarised. ## discussion from the results of this case study, quite a substantial variation in selecting gender-fair language for post-editing could be observed. when required to use gender-neutral rewording, participants omitted pronouns or repeated the character's name to avoid gender marking. nevertheless, the annotated post-edited segments also show a large degree of creativity with different rewording and terms used, e.g. "aus dem schauspiel-und musikbereich kommend" (who comes from music and acting). when instructed to use gender-inclusive characters, the majority of the participants opted for gender star (*). however, its realisation was inconsistent in the case of pronouns and articles, at times erroneous. this included the use of other characters, such as slash (/), and a different order of male and female forms. when required to use neosystems, two trends could be observed: participants either opted for more sophisticated neosystems (the sylvain system) or for easier ones (nona system and ens forms). while we ensured that the texts where equivalent in length and complexity, the fact that the neosystems came last after already two previous post-editing tasks could potentially have impacted the results. in the future, reording the sequence between participants could account for this factor. the variation in the use of strategies will probably always occur, since even if used correctly there are many ways to reword a phrase. in terms of times, the use of different strategies did not impact pe speed. the great differences in time depended on the person more than the specific gender-fair strategy. as regards perception, participants rated gender-inclusive characters as the easiest strategy, followed by rewording. nevertheless, genderneutral rewording requires considerable creativity which is sometimes perceived as challenging. participants indicated a preference for a mix between rewording and gender-inclusive characters. there was general consent that neosystems are the most difficult approach to gfl as they are largely unknown and hence feel like a foreign language, which requires practice. this was also confirmed by the occurrences of mistakes found in the post-edited translations. participants were also interviewed on whether they consider mt in combination with pe as a viable option for producing gender-fair translations. as a general response, the mt draft was considered of good quality, requiring mostly stylistic adaptations, and pe was considered less time consuming than translation from scratch. furthermore, the existing draft allowed for a focused revision of gender references. nevertheless, half of the participants stated they would not integrate mt in their workflow due to a negative view on the technology that, in their opinion, still requires extensive post-editing. in a nutshell, the results of this case study suggest that even though unable to process genderfair language, mt can still be a useful instrument for the translation of texts in which non-binary individuals are mentioned. thus, we argue that post-editing might be a faster and viable option to generate test sets for gender-fair mt than producing translations from scratch. moreover, even though differences in temporal efforts were not found among the strategies: (i) there is a tendency for longer pe times when neosystems are used which, in this study, is not statistically significant. this could be due to the small sample of participants, thus further experiments would be needed to shed light on this phenomenon; (ii) temporal effort does not necessarily correspond to the participants' perceived cognitive effort which was generally high, especially for neosystems. in terms of methodology, interesting results could be obtained with the proposed mix of methods. however, to provide a less subjective evaluation of cognitive load, eye-tracking and keylogging experiments could be a potential alternative. it should also be noted that the group of participants had an overall positive attitude to genderfair language, given that the vast majority already actively used it in their daily life and work. a repetition of the experiment with a larger, more varied population might lead to quite different results. the results suggest substantial variation in the type of gender-fair language selected by translators, even if already restricted to a specific subtype. this has implications for mt in two regards. first, gender-fair translations or post-edited translations as future input texts might vary considerably in their gender references when describing non-binary individuals and mt should be able to handle these across languages. second, gender-neutral mt as advocated by @xcite might not be the ideal option for all languages, since a clear preference for other strategies was stated by all participants in this study. ## conclusion in this first gender-fair post-editing study, professional translators revised machine translated texts containing references to non-binary individuals from the notional gender english to the grammatical gender german. substantial variation in the implementation of the three gender-fair language strategies could be observed among participants, which implies for mt that a large variety of potential gender-neutral rewording and/or use of genderinclusive characters, the two preferred strategies, need to be handled by the systems. the third strategy of utilising neosystems was perceived as requiring the highest temporal and cognitive effort. testing the cognitive and temporal load as well as success of using gfl on a larger scale and in different language pairs might be an interesting extension of the present study. for instance, eyetracking would allow for a more detailed, objective analysis of the cognitive load of each strategy. furthermore, a large-scale study across natural languages and their respective gender-fair language strategies would be interesting, especially when comparing post-editing to translation from scratch. this comparison could provide further insights into the effectiveness of post-editing within the context of gender-fair language use in the translation process.