A Taxonomy of Bias-Causing Ambiguities in Machine Translation (2022)

Abstract: This paper introduces a taxonomy of phenomena which cause bias in machine translation, covering gender bias (people being male and/or female), number bias (singular you versus plural you) and formality bias (informal you versus formal you). Our taxonomy is a formalism for describing situations in machine translation when the source text leaves some of these properties unspecified (eg. does not say whether doctor is male or female) but the target language requires the property to be specified (eg. because it does not have a gender-neutral word for doctor). The formalism described here is used internally by a web-based tool we have built for detecting and correcting bias in the output of any machine translator.

## introduction: phenomena described by the taxonomy the taxonomy we are going to introduce in this paper is based on the assumption that biased translations are always the result of unresolvable ambiguities in the source text. we will start by demonstrating on a few examples what exactly we mean by ambiguity, what makes ambiguities resolvable or unresolvable, and how the unresolvable ones inevitably lead to biased translations. this will serve as an informal introduction before we proceed to a more formal specification of everything in the rest of the paper. when translating a sentence such as she is a doctor from english into a language such as german which has no gender-neutral word for doctor, the translator (machine or human) can translate doctor either as male arzt or as female ärztin. the word doctor is ambiguous for the purposes of this translation. however, the presence of the female pronoun she should be enough to tip any well-trained machine translator towards the female reading and to translate doctor as ärztin -as indeed most of the major machine translators such as google translate and deepl do. here, the ambiguity is resolvable from context, where by context we mean the rest of the text available to the translator. now consider a similar sentence: i am a doctor. the word doctor is as ambiguous as before, but this time the ambiguity is unresolvable from context, as there is no indication anywhere in the text whether the intended referrent of i and doctor is a man or a woman. in such a situation, the machine translator will typically decide for the male translation because that is what has been seen most often in similar contexts in its training data. this is another way of saying that the machine is making an unjustified assumption: unjustified because unsupported by anything actually present in the text being translated. there are two possible ways to "read" the ambiguous word doctor, but translations produced by this machine will be consistently biased in favour of the male reading whenever context allows both readings. unresolvable ambiguities do not simply happen arbitrarily and unexpectedly. many kinds of unresolvable ambiguities tend to happen regularly and predictably when certain words occur in the source text inside certain lexicogrammatical patterns, for example i am a. . . or you are a. . . followed by a gender-neutral noun known to have two genderspecific translations in the target language. fairslator is a tool which detects such patterns and acts on them: it asks the human user to disambiguate (eg. to tell us whether they want the male or female reading) and then it re-inflects the translation accordingly. to enable all this functionality, fairslator has inside itself a taxonomy for describing how the source text is ambiguous and which way the human user wants the ambiguity to be resolved. the taxonomy describes the following kinds of unresolvable ambiguities: this covers the well-known case of "occupation words" such as doctor, teacher, cleaner, as well as some less well-known cases such as predicatively positioned adjectives in romance languages (eg. english i am @xmath0, je suis heureuse female) and verbal participles in slavic languages (eg. english i wanted @xmath1, já jsem to chtěla female). that said, the taxonomy in its current incarnation, as presented in this paper, is oriented towards translation from english into other, mainly european, languages, and there is a version of the taxonomy for each directed language pair: one for english-to-german, one for english-to-czech and so on. we can now proceed to a more formal definition of what we mean by bias. when we consider machine translation as a black box and simply take its input and output as a pair of texts (the source text in the source language plus the translation in the target language), then we can define the following concepts: unresolvable ambiguity a portion of the source text contains an unresolvable ambiguity if, in order to translate it successfully into the target language, some semantic property of it needs to be known (such as its gender or grammatical number or level of formality) but this property is not expressed in the source text and cannot be inferred from anything in the source text. unjustified assumption an unjustified assumption is what happens when, in the face of an unresolvable ambiguity, the machine translator decides for one particular reading of the ambiguous expression over others. the assumption is unjustified because nothing actually present in the source text justifies it. the machine's decision is either random or, if the translator has been constructed through machine learning, predetermined by which reading has been observed more often in the training data. bias a machine translator is biased if, while dealing with unresolvable ambiguities and deciding which unjustified assumptions to make, its decisions are not random: it makes certain unjustified assumptions more often than others. for example, if a translator consistently decides for male readings of doctor or for singular informal readings of you (when these are unresolvably ambiguous in the source text), then the translator is biased. in other words, we define bias as a purely technical concept, as the tendency of an automated system to make certain unjustified assumptions more often than others. this differs from the popular commonsense understanding of the word bias which, in addition to the purely technical process, implies harmful and unjust consequences. this implication is not a necessary part of our definition. our definition of bias covers bias regardless of whether it is harmful to society (eg. because it perpetuates a stereotype by speaking about doctors as if they must always be men), harmful to an individual (eg. because it offends somebody by addressing them with an inappropriately informal pronoun) or relatively harmless and merely factually incorrect (eg. because it addresses a group of people with a singular pronoun). interestingly, our definition applies not only to machines but also to humans: it is not unheard of for human translators to make the same kind of unjustified assumptions and to go about it with the same amount of bias as machines. good human translators avoid bias by observing the extralinguistic reality (simply looking to see eg. whether the speaker seems male or female) and by asking follow-up questions ("what do you mean by you?"). machine translators do not normally have the means to do such things but fairslator is a plugin which adds the latter ability to any machine translator: the ability to recognize unresolvable ambiguities, to ask follow-up questions, and to re-inflect the translation in accordance with the answers, in a fashion similar to @xcite . ## how fairslator uses the taxonomy the main purpose of the taxonomy is to make it possible for fairslator to formulate human-friendly disambiguation questions for users. 5 here are some examples of descriptors and the disambiguation questions generated from them. a group containing at least one man (selected) a group of women 2. st|sv|p : st ## summary machine translation technology is getting better all the time at resolving ambiguities from clues in the context. but some ambiguities can never be resolved in this way because there are no clues in the context. to avoid bias during the translation of texts that contain unresolvable ambiguities, we need to build tools which are able to (1) recognize that an unresolvable ambiguity has occured and (2) ask the human user to disambiguate manually. to be able to build such tools at all, what we need first of all is an expressive formalism for describing unresolvable ambiguities. this paper has shown how to construct such a formalism for any directed language pair by analysing the source text and its translation from the point of view of three axes (speaker, listener and bystander) and by describing any unresolvable ambiguities that occur in those axes through descriptors which tell us (1) which readings are allowed by the source text and (2) which one of those readings is actually expressed in the translation.