The Role of Compounds in Human vs. Machine Translation Quality (2023)

Abstract: We focus on the production of German compounds in English-to-German manual and automatic translation. On the example of WMT21 news translation test set, we observe that even the best MT systems produce much fewer compounds compared to three independent manual translations. Despite this striking difference, we observe that this insufficiency is not apparent in manual evaluation methods that target the overall translation quality (DA and MQM). Simple automatic methods like BLEU somewhat surprisingly provide a better indication of this quality aspect. Our manual analysis of system outputs, including our freshly trained Transformer models, confirms that current deep neural systems operating at the level of subword units are capable of constructing novel words, including novel compounds. This effect however cannot be measured using static dictionaries of compounds such as GermaNet. German compounds thus pose an interesting challenge for future development of MT systems.

## introduction assessing the quality of machine translation is a challenging task regularly tackled, e.g., in the manual evaluation of wmt translation task @xcite or in wmt metrics task @xcite . various evaluation methods have been developed for this purpose. manual evaluation in wmt has evolved from fluency and adequacy @xcite to direct assessment (da, @xcite or mqm @xcite . automatic evaluation is on the move from string matching techniques like bleu @xcite or chrf @xcite to embedding-based methods like comet @xcite or prism @xcite . none of these approaches is particularly sensitive to specific subtle phenomena such as the presence or absence of compound words, a particular grammatical construction that is frequent in german. this paper focuses on german compounds, and how they occur in human and machine translations from english. german has a highly productive word formation system mainly through compounding and derivation, especially for nouns @xcite . in this paper, we study german nominal compounds, which mostly consist of two constituents that are either complex or simple stems. the compounds in german are right-headed which means that the second element determines the morphosyntactic properties of the formed word. additionally, semantically empty elements, called linking elements, can be added to the first stem of the compound @xcite . using compounds instead of multi-word expressions is a soft phenomenon related to text style, which can affect the perceived quality of the text. native speakers regularly form new compound words to fulfill the needs requested by a particular dialogue or discourse situation. we believe that machine translation systems, operating on subword units, are able to produce complex words like humans, even if they were not included in the training data. we know that splitting and determining german compounds is a complex task. therefore, we relied on a list of compounds extracted from the german adaptation of wordnet called germanet @xcite . operating on a closed list of compounds may provide an advantage for the analysis. considering that the use of compounds is a stylistic matter, the exact list provides us with the possibility to group the observations of the phenomenon. in the paper, we study several aspects of the data and models concerning the production of german nominal compounds. ## related work most of the previous work on mt dealing with german compounds was done in the "classical" statistical machine translation (smt). we found only a few papers, see below, about german compounds in neural machine translation (nmt), almost all of which were published before the introduction of the transformer model @xcite , the current state of the art. our work focuses on the production of german compounds in transformer models, a topic that has not been adequately studied yet. ## compounds in mt outputs in our analysis, we primarily rely on compounds that are contained in the germanet list and search for them in wmt21 translations. we compare the counts of compounds found in reference translations and state-of-the-art system outputs. we present counts of compounds and sentences containing at least one compound for each reference and output translation separately. we also report the number of compounds and sentences with compounds confirmed in one or more of the reference translations. the results are sorted by the decreasing number of found compounds and listed in considering only the number of produced compounds, the best mt system is the constrained system nemo, with 842 compound occurrences in 559 sentences (see it is important to note that the same concept can be translated using various compounds, so even when the mt output contains a correct compound, it need not be confirmed by the reference. we mitigate this issue by considering four different human translations instead of only one, and also by reporting the number of sentences in which any compound appeared. ## conclusion we examined the production of german compounds in transformer models in english-to-german mt. our analysis revealed that reference translations consistently contain more compounds than mt systems. we confirmed that transformers have the ability to generate new words including compounds but evaluating compound production using closed lists or existing general manual evaluation methods (da, mqm) is not effective. this opens space for further exploration of compound production as well as their evaluation.