A Natural Diet: Towards Improving Naturalness of Machine Translation Output (2022)

Abstract: Machine translation (MT) evaluation often focuses on accuracy and fluency, without paying much attention to translation style. This means that, even when considered accurate and fluent, MT output can still sound less natural than high quality human translations or text originally written in the target language. Machine translation output notably exhibits lower lexical diversity, and employs constructs that mirror those in the source sentence. In this work we propose a method for training MT systems to achieve a more natural style, i.e. mirroring the style of text originally written in the target language. Our method tags parallel training data according to the naturalness of the target side by contrasting language models trained on natural and translated data. Tagging data allows us to put greater emphasis on target sentences originally written in the target language. Automatic metrics show that the resulting models achieve lexical richness on par with human translations, mimicking a style much closer to sentences originally written in the target language. Furthermore, we find that their output is preferred by human experts when compared to the baseline translations.

## introduction machine translation has made tremendous progress in recent years with the advent of neural methods @xcite . this is especially true for language pairs with a large amount of available bilingual text for training @xcite . however mt output still can be improved: it currently trails human translators in expert evaluation @xcite and its language is perceived as poorer and more synthetic @xcite . in this work, we aim to produce machine translation output that has a more natural style. although difficult to define precisely, we consider a translation to be natural if it is an adequate source es wird befürchtet, dass die opferzahlen noch deutlich in die höhe gehen. translationese it is feared that the number of victims will increase significantly. natural it is feared that the death toll will rise significantly. and fluent translation, whose style matches that of high quality monolingual text. such a translation should contain few translationese constructs and use a rich vocabulary. this is exemplified in figure 1 . the translationese sentence uses the construct "number of victims", which is a literal translation for the german "opferzahlen". although correct (i.e. adequate and fluent), "death toll" shows a much more natural word choice for this translation. our objective in this paper is to study how the naturalness of machine translation output can be improved. in particular, we focus on how available measures can guide the translation process towards this goal. there have been several studies analyzing the naturalness of generated texts (see section 2), but in contrast we concentrate on actively improving this aspect by modifying how nmt output is produced. our methodology follows a simple intuition: training data whose target side resembles highquality text naturally written in the target language can bring model outputs closer to this style of text. we exploit the fact that bilingual training sets typically mix examples originating from both translation directions: source-to-target and targetto-source. we rely on contrasting language models (lms) @xcite to identify natural data: we train sep-arate models on target-language data known to be translations, and on data known to be mostly originally written in the target language. we then use these lms to tag parallel training data as having a natural or translated target side. comparing to hard filtering of the data, tagging offers more flexibility without sacrificing coverage @xcite . our contributions are as follows: (1) we use contrastive language model scoring to separate natural from translated text. (2) we demonstrate that optimizing bleu scores on tgt-original test sets while avoiding high bleu scores on src-original test set is a valid strategy to improve the naturalness of mt output. (3) we show that our more natural mt output is more similar to natural sentences based on lexical diversity. (4) human evaluations show that the style of our more natural translations are preferred by humans, albeit with a minimal loss in translation accuracy. ## approach our first objective is to distinguish text originally written in the target language (natural text) from translations. for that purpose, we train a pair of sentence-level language models to contrast their likelihood, a proven method for domain adapta-tion @xcite . these language models are then used to tag mt training data as natural target (<@xmath0, in order to train an mt system which can favor natural hypotheses. ## experimental setup we experiment on the wmt news translation tasks for evaluation @xcite , focusing on the german↔english language pair. for this language pair there is abundant training data available, and mt systems achieve high quality translations. this is a good setting for our work since improving naturalness becomes a worthwhile endeavor only if high accuracy and fluency levels are reached. ## experimental results due to space constraints and german being the more morphologically rich language, we focus our analysis mainly on the @xmath1, but we provide translation results for the reverse direction (@xmath2. ## conclusion we propose a method for achieving more natural translations, i.e. translations which adopt a style closer to text originally written in the target language. using contrastive language model scoring we classify our training data depending on whether the target side was originally written in the target language or whether it is a translation. this information is given to the translation system via an input tag, so that we can bias the generation process towards producing output closer to natural text. we demonstrate that building an mt system focusing on natural translations can be evaluated by optimizing bleu on target-original test sets while avoiding high bleu scores on src-original test sets. through automatic metrics we show that the n2n method achieves lexical diversity closer to that of natural sentences indicative of more natural text. indeed, human evaluations show that the produced translations are preferred by human judges when asked to choose the more natural translation. there is some drop in translation accuracy, as shown by the mqm analysis, however this can be an acceptable trade-off for some applications. for example, when considering post-editing, a more natural initial proposal will most certainly result in a more natural final output, while accuracy errors are usually easier to detect and fix for human post-editors. the main contribution of this work lies in highlighting the potential for more natural translations by appropriate manipulation of the training data and evaluation measures. our approach for using this information through tagging is a good first step, but it is a straightforward data manipulation. other techniques that modify the model architecture or training objective may allow us to achieve the same improvements in naturalness without loss in translation accuracy. source es wird befürchtet, dass die opferzahlen noch deutlich in die höhe gehen. baseline it is feared that the number of victims will increase significantly.