Test Suite Evaluation: Morphological Challenges and Pronoun Translation (2022)

Abstract: This paper summarizes the results of our test suite evaluation with a main focus on morphology for the language pairs English to/from German. We look at the translation of morphologically complex words (DE–EN), and evaluatewhether English noun phrases are translated as compounds vs. phrases into German. Furthermore, we investigate the preservation of morphological features (gender in EN–DE pronoun translation and number in morpho-syntacticallycomplex structures for DE–EN). Our results indicate that systems are able to interpret linguistic structures to obtain relevant information, but also that translation becomes more challenging with increasing complexity, as seen, for example, when translating words with negation or non-concatenative properties, and for the morecomplex cases of the pronoun translation task.

## introduction evaluating mt output is challenging. documentlevels metrics give a rather coarse-grained estimation of the overall translation quality, but cannot determine how well a system operates for particular challenges. translations do not have a deterministic solution, but there are always several possibilities for a valid translation, making a focused evaluation of particular phenomena difficult. the annual wmt shared task provides the possibility to submit custom test suites to be translated in addition to the regular test sets, which allows the investigation of the translation performance of state-of-the-art systems when presented with particular translation tasks. in this test suite, we focus on morphological challenges for english to/from german translation: for german-english translation, we look at the translation of morphologically complex words, in addition to a small set of sentences where a subtle difference (singular vs. plural) needs to be detected. for english-german, we study how complex noun phrases are translated -as compounds or rather as multi-word phrases. furthermore, we add a pronoun translation task and evaluate the translation of the english pronoun it into its german equivalents er/sie/es, depending on the gender of the noun it refers to. the test suite does not aim at measuring a system's general translation performance -this is already assessed by means of a manual evaluation and various other metrics in the main shared taskbut rather at evaluating the translational behaviour for carefully selected words or phrases. as the sentences in the test suite are not parallel, we opt for a semi-automatic approach where translation options for the words in question are manually collected and then matched with the translation output. thus, only the translation of the relevant word is considered, whereas the rest of the sentence is ignored. ## data creation and evaluation in the following, we outline the process of composing and evaluating the test suite. ## de-en translation this section summarized the design and the outcome of the four categories in de-en translation. ## en-de translation in this section, we look at re-translating compounds and present a pronoun translation task. ## related work the linguistic and morphological compentence of translation systems is a topic of previous and ongoing research. @xcite present a challenge set for english to french translation targeting linguistic divergence between the two language pairs. their hand-crafted set has a focus on morpho-syntactic, lexico-syntactic and syntactic divergences. @xcite present an analysis of minimal pairs representing a contrast that is expressed syntactically in en and morphologically in a morphologically rich language (de, cz and lv). for a source test sentence (the base), variant(s) containing exactly one difference with the base (e.g. person/number/tense of a verb or number/case of a noun/adjective or polarity) are generated and automatically evaluated, counting a translation as correct if the targeted feature is produced correctly in the target language. the work of @xcite and @xcite comprises the dfki test suite for german to english mt. their test set consists of over 5k sentences to analyze over 100 categories, including negation, composition, function words, subordination, non-verbal agreement, multi-word expressions, verb tense/aspect/mood, lexical ambiguity and punctuation. lingeval97 @xcite is a large-scale data set of 97000 contrastive english-german translation pairs where errors (on the level of agreement, auxiliaries, verb particles, polarity and swapped compound components) haven been automatically created. it is then measured whether a reference translation is more probable than the corresponding contrastive translation containing an inserted error. an obvious question with this approach is whether forced translation mimics the mt system's "natural behaviour", i.e. whether the presented sentence e is the system's best choice given the source sentence f. this question is addressed in @xcite where it is argued that test data should be chosen such that there is minimal discrepancy between the training data and the data to be evaluated. they recommend that test sentences be created from machine generated text rather than using human-written references. the paper proposes an updated version of lingeval97. ## conclusion and future work this paper summarizes the results of our wmt22 test suite, looking at the translation of morphologically complex words, compounds and a set of minimal pairs to assess the preservation of number. our evaluation shows that on one side, the translation of morphologically complex words is not without challenges, in particular for low-frequency words and when containing negation. on the other hand, the handling of the (structurally much simpler) compounds nn1 nn2 vs nn2 nn1 and the preservation of the number feature worked quite well. the results for the pronoun translation experiment were mixed. our results indicate that the systems have a generally good understanding of linguistic structures, but also that at a certain degree of (morphological) complexity, problems start to arise. for research in mt, this means that modeling morphology, particularly negation and non-concatenative processes, might be worthwhile. the test suite, with the exception of the pronoun translation task, is based on a manually created set of sentences alongside matching dictionaries. while this has the advantage of presenting the selected words/phrases in a natural context, it comes with a comparatively high amount of manual effort, making it difficult to upscale. in contrast, the artificial data used in the pronoun translation task allows for a comparatively straightforward evaluation, but sounds unnatural and likely differs considerably from the mt training data, which might even bias the results to a certain extent. for future work, we intend to look into the generation of meaningful sentences with particular properties that allow for a systematic evaluation of mt.