On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation (2022)

Abstract: Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU. We compare translations from supervised and unsupervised MT systems of similar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is supervised MT. We then demonstrate a way to combine the benefits of both methods into a single system which results in improved adequacy and fluency as rated by human evaluators. Our results open the door to interesting discussions about how supervised and unsupervised MT might be different yet mutually-beneficial.

## introduction supervised machine translation (mt) utilizes parallel bitext to learn to translate. ideally, this data consists of natural texts and their human translations. in a way, the goal of supervised mt training is to produce a machine that mimicks human translators in their craft. unsupervised mt, on the other hand, uses monolingual data alone to learn to translate. critically, unsupervised mt never sees an example of human translation, and therefore must create its own style of translation. unlike supervised mt where one side of each training sentence pair must be a translation, unsupervised mt can be trained with natural text alone. in this work, we investigate the output of supervised and unsupervised mt systems of similar quality to assess whether systematic differences in translation exist. our exploration of this research area focuses on @xmath0, allowing us * work completed at google translate research. our main contributions are: our results provoke interesting questions about what unsupervised methods might contribute beyond the traditional context of low-resource languages which lack bilingual training data, and suggest that unsupervised mt might have contributions to make for high-resource scenarios as well. it is worth exploring how combining supervised and unsupervised setups might contribute to a system better than either creates alone. we discuss related work in §2. in §3, we introduce the dataset, model details, and evaluation setups. in §4, we characterize the differences between the output of unsupervised and supervised neural mt systems of similar quality. in §5, we demonstrate a combined system which benefits from the complementary strengths of the two methods. we summarize the paper in §6. monolingual data in mt bt is widely-used to exploit monolingual data @xcite . "semi-supervised" systems use monolingual and parallel data to improve performance (e.g. @xcite ). @xcite combine multilingual supervised training with mass for many languages and zero-shot translation. source artifacts in translated text because supervised mt is trained ideally on human-generated translation, characteristics of human translation affects the style of machine translation from such systems. dubbed "translationese," human translation includes source language artifacts @xcite and source-independent artifacts-translation universals @xcite . there are systematic biases inherent to translated texts @xcite , and biases coming from interference from source text @xcite . in mt, @xcite attribute these patterns as a source of mismatch between bleu @xcite and human evaluation measures of quality, raising concerns that overlap-based metrics reward hypotheses with the characteristics of translated text more than those with natural language. @xcite note loss of linguistic diversity and richness from mt, and @xcite see related effects even after human post-editing. the impact of translated text on human evaluation has also been studied @xcite @xcite @xcite , as has the impact in training data @xcite @xcite . measuring word reordering word reordering models are well-studied because they formed a critical part of statistical mt (see @xcite for a review). others examined metrics for measuring reordering in translation (e.g. @xcite @xcite . @xcite and @xcite use part-of-speech (pos) tags in the context of parse trees, and @xcite measure the similarity of french and english with respect to phrasal cohesion by calculating alignment crossings using parse trees. most similar to us, @xcite view simplified word alignments as permutations and compare distance metrics over these to quantify the amount of reordering done. they use ter computed over the alignments as a baseline. @xcite 's lrscore interpolates a reordering metric with a lexical translation metric. ## application: leveraging unsupervised back-translation our results indicate that high-adequacy unsupervised mt output is more natural and more structurally diverse in comparison to human translation, than is supervised mt output. we are thus motivated to use these advantages to improve translation. we explore how to incorporate unsupervised mt into a supervised system via back-translation. we train for ∼500,000 updates for each experiment, and select models based on validation performance on newstest2018. we test on newstest2019(p). ## conclusion recent unsupervised mt systems can reach reasonable translation quality under clean and controlled data conditions, and could bring alternative translations to language pairs with ample parallel data. we perform the first systematic comparison of supervised and unsupervised mt output from systems of similar quality. we find that systematic differences do exist, and that high-quality unsupervised mt output appears more natural and more structurally diverse when compared to human translation, than does supervised mt output. our findings indicate that there may be useful differences between supervised and unsupervised mt systems that could contribute to a system better than either achieves alone. as a first step, we demonstrate an unsupervised back-translation augmented model that takes advantage of the differences between the translation methodologies to outperform a traditional supervised system on human-evaluated measures of adequacy and fluency.