Identifying Context-Dependent Translations for Evaluation Set Production (2023)

Abstract: A major impediment to the transition to contextual machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated correctly are rare in test sets, reducing the utility of standard corpus-level metrics such as COMET or BLEU. On the other hand, datasets that annotate such sentences are also rare, small in scale, and available for only a few languages. To address this, we modernize, generalize, and extend previous annotation pipelines to produce MultiPro, a tool that identifies subsets of parallel documents containing sentences that require context to correctly translate five phenomena: gender, formality, and animacy for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input to the pipeline is a set of hand-crafted, per-language, linguistically-informed rules that select contextual sentence pairs using coreference, part-of-speech, and morphological features provided by state-of-the-art tools. We apply this pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT, and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its performance using both overlap with previous work and its ability to discriminate a contextual MT system from a sentence-based one. We release the MultiPro pipeline and data as open source.

## introduction neural machine translation (nmt) systems can produce high-quality, fluent output which are nearly indistinguishable from human translations, when evaluated at the sentence level. this humanlevel parity has been shown to disappear, however, when evaluated in context @xcite . this is unsurprising, because sentences are nearly always written by humans in some contextual setting, and are translated by translators in the same fashion. dismissing this context translation to another language must address ambiguities where the semantic or grammatical granularity of two sentences is imbalanced or mismatched. probably the most widely-known of these is grammatical gender, i.e., when translating referential pronouns from a grammatically nongendered language to a gendered one. for example, when translating from english to french, the pronoun it must be translated to il or elle depending on the grammatical gender of the antecedent noun, which may not be available in the same sentence. the obvious path forward in addressing these issues is to move to contextual machine translation, in which sentences are no longer translated in isolation but with their source-side context. recent work has shown that transformers @xcite are capable of handling longer sequences and improving performance on context-based evaluation @xcite . however, general contextual translation has english target auxiliary i just figured you need to know. and now you do. et maintenant tu sais. i can't lose my voice. you won't. ## contextual phenomena a number of context-based phenomena which create ambiguities are common. we display some examples in we describe each phenomena for comprehension and then provide our extraction methodology in order to identify when these ambiguities arise. deep within the royal vault there is a statue called the palladium. ## extraction pipeline our pipeline functions by identifying up to four key tokens and ensuring each token matches a set of predefined criteria. the four components are: (1) the source (english) token defined as 𝑇 𝑠 , the target (non-english) token defined as 𝑇 𝑡 , the source token which conveys the contextual information required to resolve the ambiguity defined as 𝐶 𝑠 , and the target token aligned to 𝐶 𝑠 defined as 𝐶 𝑡 . these relationships are illustrated in figure 1 . contextual information is defined by a contextual relationship, 𝑄, which has an associated solver. the predefined criteria is a set of rules, 𝑅. we can identify ambiguous sentences by: each aligned pair of words forms a potential 𝑇 𝑠 -𝑇 𝑡 pair. 2. ensure 𝑇 𝑠 meets all criteria 𝑅 𝑇 𝑠 3. ensure 𝑇 𝑡 meets all criteria 𝑅 𝑇 𝑡 4. apply a solver for the contextual relationship, 𝑄 to the english token 𝑇 𝑠 and its preceding context to identify 𝐶 𝑠 . 5. ensure 𝐶 𝑠 meets all criteria 𝑅 𝐶 𝑠 . 6. identify the target token 𝐶 𝑡 via word alignment to 𝐶 𝑠 . if translation conveys semantic symmetry, this token also has a contextual relationship with 𝑇 𝑡 . 7. ensure 𝐶 𝑡 meets all criteria 𝑅 𝐶 𝑡 consider the ambiguity of pronoun resolution. @xcite first proposed a pipeline for extracting ambiguous translations of english "it" to german nominatives ("er", "es", and "sie"). we can explain their methodology foot_1 via the aforementioned definition. the following identifies all ambiguities where the english "it" is translated as "sie." 1. for each source-target sentence pair, apply word alignment. each aligned pair of words forms a potential 𝑇 𝑠 -𝑇 𝑡 pair. the same criteria could be enumerated for the masculine and neuter equivalents, appropriately changing gender and surface form checks. to extract a specific phenomenon and language, a "rule" (𝑅) must be written which specifies features that 𝑇 𝑠 , 𝑇 𝑡 , 𝐶 𝑠 , and 𝐶 𝑡 must have. these features can range from part-of-speech, lemma, gender, case, plurality or others. the manner in which gender following previous works, we retrieve 𝑇 𝑠 and 𝑇 𝑡 based on surface form and word alignment. 𝐶 𝑠 is a noun discovered via coreference chain. if the coreference is a noun phrase, the head of the phrase is used. 𝐶 𝑡 is retrieved via word alignment. 𝐶 𝑡 must match the same morphological features present in 𝑇 𝑡 (e.g., gender and number). animacy as explained in section 2.1.2, the animacy ambiguity that we consider occurs when translating from the gendered languages into english (whereas the gender ambiguity occurs when translating out-of english). to extract these examples, we use the same rules as gender, but we reverse the language direction for inference. ## quantitative evaluation our goal is to show that our test sets can usefully discriminate between sentence-level and contextaware systems. an impediment to this goal is the lack of contextual machine translation models across languages for use in comparison and evaluation, and the difficulty in building them. consequently, we turn to a commercial system, deepl, which is alone among commercial providers in advertising contextual translation. 5 we translate with document-context by providing deepl with context when translating, and compare to the same model translating without context at the sentence level. we show that a contextual system appropriately benefits from the additional context and gains significance performance on this test set. many works release their evaluation sets with the assumption of contrastive evaluation @xcite @xcite , where the test is whether a model assigns a higher score to correct data than to linguistically-manipulated counterparts. this assumption ignores the fact that machine translation is a generative problem and should be evaluated as such. recent work @xcite we validate our data by showing it (1) adequately addresses context-based phenomena and (2) is sufficiently challenging. we demonstrate the former by showing that a context-aware translation model consistently outperforms a context-less equivalent. we see the latter is true as the contextual model does not solve the problem. there is still significant context-aware work to be done. ## qualitative evaluation our extraction pipeline relies on handbuilt rules applied to the outputs of automatic tools. as a result, the process is noisy and may be susceptible to errors. the previous section showed that a contextual system does better on our test sets than its sentencebased counterpart, and there is no reason we can think of to suspect that errors would systematically benefit the contextual system. however, in the interest of completeness, we took a more qualitative look at the data. this includes a systematic manual review ( § 5.1), direct comparison with prior work ( § 5.2), and an error analysis ( § 5.3). ## analysis of wmt test sets as previously earlier, this pipeline is easily applied to new data and test sets. we demonstrate this by applying it to the 2019-2022 wmt newswire test sets @xcite @xcite . in so doing, we find ## related work work in contextual machine translation can be divided into three categories: (1) the publication of resources, similar to this work; (2) alterations on the training paradigm via architecture or data input; a substantial amount of work has been done to allow traditional neural models to handle additional input. some approaches involve more complex architectures or modifications to training paradigms incorporate longer sequences @xcite , but @xcite showed that unaltered transformers can handle longer sequences. other work has focused on leveraging and cleaning the available data, since large-scale document bitext is lacking @xcite . lastly, many have realized that bleu, comet, or other sentence-level metrics will not address the distinction in document-level performance. @xcite proposed a new method for adjusting current methods to adjust for documentlevel inputs. @xcite proposed blonde, an entirely novel metric for document-level evaluation. we hope this work complements these works and serves to further the field in its aspirations towards true context-aware translation. ## summary machine translation systems face a performance ceiling that can't be overcome so long as they continue to operate at the sentence level. a major obstacle to that transition is the unavailability of test sets in many languages and for many contextual phenomena. the goal of this work has been to help address that problem. the extraction pipeline proposed in this paper can be used to identify and generate new test sets which contain linguistic phenomena that can only be consistently translated by contextual systems. the application of our pipeline to the opensubtitles dataset in seven languages provides a new set of evaluation sets including a wider set of languages and phenomena than were available before. further, we hope that the extensibility of our pipeline to new phenomena and languages allows for others to build upon this work to expand resources and coverage. the ctxpro datasets and extraction pipeline are available as open source from https://github.com/rewicks/ctxpro .