Context-aware and gender-neutral Translation Memories (2023)

Abstract: This work proposes an approach to use Part-Of-Speech (POS) information to automatically detect context-dependent Translation Units (TUs) from a Translation Memory database pertaining to the customer support domain. In line with our goal to minimize context-dependency in TUs, we show how this mechanism can be deployed to create new gender-neutral and context-independent TUs. Our experiments, conducted across Portuguese (PT), Brazilian Portuguese (PT-BR), Spanish (ES), and Spanish-Latam (ES-LATAM), show that the occurrence of certain POS with specific words is accurate in identifying context dependency. In a cross-client analysis, we found that ~10% of the most frequent 13,200 TUs were context-dependent, with gender determining context-dependency in 98% of all confirmed cases. We used these findings to suggest gender-neutral equivalents for the most frequent TUs with gender constraints. Our approach is in use in the Unbabel translation pipeline, and can be integrated into any other Neural Machine Translation (NMT) pipeline.

## introduction translation memory (tm) servers are dynamic databases that store frequent and high-quality translated segments @xcite . in the context of computer-aided translation (cat) tools and in the machine translation (mt) industry, these segments are used as complementary modules, to optimize the translation process, reducing costs and increasing the speed of translations. a downside of these systems is that they are usually based on sentence-like units. combining the sentences risks coherence problems at the document level. this is problematic as a high-quality text is a coherent and cohesive unit with ideas presented in a logical way. recent work in this field aims to achieve a context-aware mt system by incorporating more context than the current sentence and tackling discourse phenomena across the document. our use case is customer-support content: a content type that differs significantly from other types of text since it represents an interaction between an agent and a customer, usually over email or chat messages. this implies that the text contains a lot of first and second person, and pronouns or anaphora. even third person references can also be gendered in the target languages in question (e.g., "está encantada?" "está encantado?" / are you thrilled?). since the translation process is carried out sentence by sentence, gender information associated with that of the addressee is easily lost, resulting in agreement problems throughout the document. we hypothesize that, by using pos information, we can automatically identify context-dependent and context-independent segments. with the results of context identification, we can then transform segments with gender constraints into gender-neutral tus, so that these can be reused without damaging translation quality through agreement errors. as such, part of our work is concerned with creating gender-neutral content. gender inclusivity is a topic of increasing concern in natural language processing (nlp) and responsible ai, and exist-ing approaches to tackle it have used neutral pronouns @xcite , or words or expressions that do not require gender-marking at all @xcite . aside from creating a more inclusive tone with customers, using gender-neutral content also allows us to reuse the same segments without limits, and without compromising meaning or quality. our goal is to show how this approach, applied here in the customer-support domain, is equally applicable to other industries and to existing nlp pipelines. similarly, our strategies to identify context patterns can be integrated into any neural machine translation (nmt) pipeline. ## related work context is pivotal to understanding a text. words are bound to fixed semantics, but can acquire distinct meanings in different contexts. one of the major issues faced in translation, language ambiguity @xcite , is linked to the innate flexibility of words and sentence structures in terms of their semantic value. context plays a fundamental role in allowing us to decode ambiguous segments. the relationship between context and linguistic expression is complex; context influences and is influenced by linguistic expression. as @xcite describes, the dependency between these two dimensions (form and the background) is omnipresent, and is decisive for the construction and recovery of meaning. in our line of investigation, we use this perspective to observe the relationship between context and the linguistic structures within a document. we consider context crucial for an accurate word/sentence interpretation that would otherwise be lost or misunderstood, resulting in ambiguities and nonsensical text. a text is a linguistic object with properties organised around discursive concepts like textual coherence and cohesion. these allow us to perceive a sequence of sentences as a unity @xcite . however, contemporary mt still very much relies on a sentence-based translation approach, where sentences are translated in isolation, disregarding context and referential dependencies within a document @xcite @xcite . such tradition can lead to inappropriate and erroneous translations, introducing ambiguities, register and gender issues, inter alia. in recent years, the shift towards document-level translation or context-aware machine translation has sought to overcome the limitation of current state-of-the-art mt models, and to solve intersentential dependencies by taking into account discursive phenomena @xcite . demonstrating the importance of context, @xcite analyzed and evaluated the performance of 16 nmt systems on the translation of pronouns from english to german with a test set with 200 pronouns. the authors found that all of the nmt systems analyzed had a better performance translating pronouns with intersentential reference. in contrast, the translation of anaphoric pronouns, whose reference was within the sentence, was more difficult. @xcite created english to french test sets that tackled coreference, lexical coherence and cohesion as context-aware categories, in order to test the performance of a nmt system with a multi-encoder architecture. the results of their experiment showed positive outcomes for both coherence and cohesion, and conversely, less favorable results for coreference. in @xcite , the author annotated an english-brazilian portuguese corpus made of 60 documents with a total of 3,680 sentences, from six different domains: literary, subtitles, news, reviews, medical and legislation. they consider gender agreement, number agreement, lexical ambiguity, reference, ellipsis and terminology as contextaware issues. the aim of the study is to use the corpus as a test set for the evaluation of mt and quality estimation and to perform linguistic analysis of context issues. in turn, @xcite , developed scat (supporting context for ambiguous translations), a set of english/french bitexts for contextual support, that access the context used for disambiguation through the identification of the position and characteristics of elements composing a referential chain, thus allowing for the creation of future models that use context effectively. to create the test set, the authors requested annotations from professional translators on the context they considered relevant to resolve intersentential ambiguities in translation. to generalize, we can state that most relevant work in the domain focuses on the same discursive parameters, such as i) coreference and anaphora resolution (tracing back the referents of previously-mentioned entities); ii) lexical cohesion (investigating how the different cohesion devices that occur beyond the sentence level are correct); and iii) discourse connectives (exploring how the translation of these words affects the interpretation of the text @xcite @xcite @xcite ). regarding our approach to use pos for gender-neutral and for minimizing context-dependent tus, to the best of our knowledge, this is the first attempt of its kind. ## methodology as previously mentioned, the main goal of context-independent translations is that they can be reused without compromising the coherence of a whole document. in our approach to automatically detect context, we started by classifying a dataset of very frequent tus from various clients as context-dependent or context-independent, assessing full documents. this dataset comprised 6,368 tus for pt; 28,604 tus for es; 33,623 tus for pt-br and 10,026 for es-latam. due to the high volume of entries, we selected only a sample of the original dataset for analysis: 1,300 entries per language-pair, for a total of 5,200. to annotate this data, we developed a context annotation typology (described in section 3.1). then, we used stanza @xcite , a pos tagger that uses the universal dependencies (ud) 1 framework of 17 tags, to identify the pos patterns associated with context-dependent segments. finally, we added 8,000 tus to the dataset for the same four language varieties, using data from seven different clients within the customer-support domain, but with distinct subject matters (such as technology and retail) in order to test the reliability of our first annotation across a range of content. this time around, we considered only segments with gender-related issues, since these were the most frequent issue type in our dataset. the goal of this analysis was to verify whether our hypothesis would hold across new content; that the pos patterns found would indeed help identify context-dependent segments. in total, we analyzed 13,200 tus according to these predefined contextdependency patterns. the number of tus for all of the experiments is displayed in ## pos patterns distribution the annotation of the 5,200 tus in the first experiment showed that 6.5% of the segments were context-dependent. as for the context-related issues, gender agreement corresponded to 98% of all cases and the remaining had residual occurrences, namely register (1.2%), terminology (0.3%), and ellipsis (0.5%). after the annotation step, we applied stanza to identify pos patterns. to do so, we used the pos category that was context-dependent, marked with an asterisk (*), and the category that precedes or follows it. as a result, we found eight patterns that are presented in out of the 8,000 analyzed segments in the second experiment, only 4,224 tus matched with one of the eight patterns found for both context-dependent and context-independent. results showed that 19% of these tus were contextdependent and 81% were context-independent. as for the eight pos patterns (see ## pos patterns and context-dependent words after the previous experiment, where we identified that the pos patterns were not exclusive of context-dependent segments, also occurring in context-independent ones, we performed a rootcause analysis to understand if there were distinctive features. while annotating, it was clear that context-dependent segments usually involved specific keywords that were also very frequent amongst all the data, such as "obrigado(a)" for pt or "entantado(a)" for es. therefore, a new analysis was conducted, in which we aimed to verify if using these frequent keywords in conjunction with the pos patterns would facilitate the detection of context-dependent segments. ## conclusions we analyzed 13,200 tus and identified over 1,298 context-dependent segments of which 1,263 had gender constraints. for these gender-constrained segments, we proposed gender-neutral alternatives by either replacing the gendered words with neutral alternatives or by syntactically manipulating the sentence in order to obtain a gender-neutral sentence with an equivalent meaning. we hypothesized that pos information would enable us to automatically identify all contextdependent segments. in addressing this, we were able to identify eight patterns: one exclusive for pt (*pron+verb), two exclusives for pt and pt-br (*verb and *verb+adp), one for es and es-latam (adp+*pron) and the remaining were common for all languages (verb+*adj, aux+*adj, verb+*pron and aux+*verb). however, our experiments with pos lead us to conclude that the pos patterns do not discriminate sufficiently between context-dependent and context-independent classification in all cases. conducting a root-cause analysis, we notice that context-dependent segments involve specific words. for instance, the 3rd person singular pronouns for pt and pt-br (-lo and -la) and 1st person plural pronouns for es and es-latam (nosotros) were very common and only knowing that these pronouns can occur is already very informative. we also analyzed adjectives such as satisfeito(a), interessado(a) or encantado(a) and emocionado(a), and other similar adjectives that allow one to express appreciation or dissatisfaction and also specific participial verb forms such as obrigado/obrigada. the frequency of this specific vocabulary in the customer-support domain may in fact tell us more than the pos information. another contribution of this work was to verify that these very frequent lemmas combined with the pos information can add value to the improvements of a system detecting context-(in)dependent tus and generating gender-neutral alternatives when possible. following our evaluation, we were able to suggest gender-neutral tus for the most frequent segments. all the segments proposed were then verified by professional linguists and translators who are native speakers of the languages in question. after review, the segments were injected into our production tm database. these segments maintain equivalent meaning but are now no longer context-dependent. the work described is applied in production at unbabel, aligned with clients' reports and nlp modules. since completing work to transform the most frequent tus analyzed into their genderneutral equivalents, we have seen a reduction in gender-agreement errors (since our translations are more consistent at the document level), and a reduction in editing time. although we demonstrate this approach in the context of the customersupport domain, we argue that it can be applied elsewhere: to other domains, industries and nlp pipelines. this work is now being used as validation and test sets for large language models assessment for gender-neutral mt.