On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation
Abstract
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions.
As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU.
We compare translations from supervised and unsupervised MT systems of similar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is supervised MT.
We then demonstrate a way to combine the benefits of both methods into a single system which results in improved adequacy and fluency as rated by human evaluators.
Our results open the door to interesting discussions about how supervised and unsupervised MT might be different yet mutually-beneficial.
Introduction
Supervised machine translation (MT) utilizes parallel bitext to learn to translate.
Ideally, this data consists of natural texts and their human translations.
In a way, the goal of supervised MT training is to produce a machine that mimicks human translators in their craft.
Unsupervised MT, on the other hand, uses monolingual data alone to learn to translate.
Critically, unsupervised MT never sees an example of human translation , and therefore must create its own style of translation .
Unlike supervised MT where one side of each training sentence pair must be a translation, unsupervised MT can be trained with natural text alone.
In this work, we investigate the output of supervised and unsupervised MT systems of similar quality to assess whether systematic differences in translation exist.
Our exploration of this research area focuses on English → German for which abundant bilingual training examples exist, allowing us ∗ to train high-quality systems with both supervised and unsupervised training.
Our main contributions are:
• We observe systematic differences between the output of supervised and unsupervised MT systems of similar quality.
High-quality unsupervised output appears more natural , and more structurally diverse when compared to human translation.
• We show a way to incorporate unsupervised back-translation into a standard supervised MT system, improving adequacy, naturalness, and fluency as measured by human evaluation.
Our results provoke interesting questions about what unsupervised methods might contribute beyond the traditional context of low-resource languages which lack bilingual training data, and suggest that unsupervised MT might have contributions to make for high-resource scenarios as well.
It is worth exploring how combining supervised and unsupervised setups might contribute to a system better than either creates alone.
We discuss related work in §2.
In §3, we introduce the dataset, model details, and evaluation setups.
In §4, we characterize the differences between the output of unsupervised and supervised neural MT systems of similar quality.
In §5, we demonstrate a combined system which benefits from the complementary strengths of the two methods.
We summarize the paper in §6.
Related Work Unsupervised MT
Two paradigms for unsupervised MT are finding a linear transformation to align two monolingual embedding spaces (Lample et al., 2018a,b; Conneau et al., 2018; Artetxe et al., 2018, 2019), and pretraining a bi-/multilingual language model then finetuning on a translation task (Conneau and Lample, 2019; Song et al., 2019;
Liu et al., 2020).
We study the Masked Sequence- to-Sequence Pretraining (MASS) language model pretraining paradigm of Song et al.
MASS is an encoder-decoder trained jointly with a masked language modeling objective on monolingual data.
Iterative back-translation (BT) follows pretraining.
Monolingual Data in MT
BT is widely-used to exploit monolingual data (Sennrich et al., 2016). “Semi-supervised” systems use monolingual and parallel data to improve performance (e.g. Artetxe et al.
Siddhant et al.
(2020) combine multilingual supervised training with MASS for many languages and zero-shot translation.
Source Artifacts in Translated Text
Because supervised MT is trained ideally on human-generated translation, characteristics of human translation affects the style of machine translation from such systems.
Dubbed “translationese,” human translation includes source language artifacts (Koppel and
Ordan, 2011) and source-independent artifacts—
Translation Universals (Mauranen and Kujamäki, 2004).
There are systematic biases inherent to translated texts (Baker, 1993; Selinker, 1972), and biases coming from interference from source text (Toury, 1995).
In MT, Freitag et al.
(2019, 2020) attribute these patterns as a source of mismatch between BLEU (Papineni et al., 2002) and human evaluation measures of quality, raising concerns that overlap-based metrics reward hypotheses with the characteristics of translated text more than those with natural language.
Vanmassenhove et al. (2019, 2021) note loss of linguistic diversity and richness from MT, and Toral (2019) see related effects even after human post-editing.
The impact of translated text on human evaluation has also been studied (Toral et al., 2018; Zhang and Toral, 2019;
Graham et al., 2019; Fomicheva and Specia, 2016;
Ma et al., 2017), as has the impact in training data (Kurokawa et al., 2009; Lembersky et al., 2012;
Bogoychev and Sennrich, 2019; Riley et al., 2020).
Measuring Word Reordering Word reordering
models are well-studied because they formed a critical part of statistical MT (see Bisazza and Federico (2016) for a review).
Others examined metrics for measuring reordering in translation (e.g. Birch et al., 2008, 2009, 2010).
Wellington et al.
(2006) and Fox (2002) use part-of-speech (POS) tags in the context of parse trees, and Fox (2002) measure the similarity of French and English with respect to phrasal cohesion by calculating alignment crossings using parse trees.
Most similar to us, Birch (2011) view simplified word alignments as permutations and compare distance metrics over these to quantify the amount of reordering done.
They use
TER computed over the alignments as a baseline.
Birch and Osborne (2011)’s LRScore interpolates a reordering metric with a lexical translation metric.
Experimental Setup 3. Data Training
Experiments are in English → German.
For the main study comparing supervised and unsupervised MT, we use News Commentary v (329,000 sentences) as parallel bitext for the supervised system, and News Crawl 2007-17 as monolingual data for the unsupervised system.
Deduplicated News Crawl 2007-17 has 165 million English sentences and 226 million German sentences.
The combined system demonstration at the end of our work utilizes a BT selection method.
We use the bilingual training data from WMT2018 (Bojar et al., 2018)
(News Commentary v13, Europarl v7,
Common Crawl, EU Press Release) so that our model can be compared with well-known work using BT (e.g. Edunov et al., 2018; Caswell et al., 2019).
We deduplicate and filter out pairs with > 250 tokens in either language or length ratio over 1.5, resulting in 5.2 million paired sentences.
Development and Test Sets
For the main experiments, we use newstest2017 as the dev set with newstest2018 and newstest2019 for test.
newstest2018 was originally created by translating one half of the test data from English → German (origen) and the other half from German → English (origde).
Since 2019, WMT produces newstest sets with only source-original text and human translations on the target side to mitigate known issues when translating and evaluating on target-original data (e.g. Koppel and Ordan, 2011; Freitag et al., 2019).
For most experiments, we evaluate on orig-en sentences only to reflect the real use-case for translation and modern evaluation practice.
We examine orig-de only for BLEU score as an additional data point of difference between supervised and unsupervised MT.
Zhang and Toral (2019) show that target-language-original text should not be used for human evaluation (orig-de, in our case).
We use the newstest2018 “paraphrased” test references from Freitag et al.
(2020), 1 which are made for orig-en sentences only.
These additional references have different structure than the source sentence but maintain semantics, and provide a way to measure system quality without favoring translations with the same structure as the source.
Observing work that uses these references, BLEU is typically much lower than on original test sets, and score differences tend to be small but reflect tangible quality difference (Freitag et al., 2020).
For the system combination demonstration, we use newstest2018 for development and newstest2019 for test.
We also use [-newstest
-]newstest
German → English and swap source and target to make an orig-de English → German test set, and use paraphrase references for newtest2019 (orig-en).
Testing on the official newstest2018 in the main experiments allows us to see interesting differences between unsupervised and supervised MT that are hidden with newstest2019 because it is orig-en only.
Using newstest2018 for development in the system combination demonstration aligns with similar literature (e.g. Edunov et al., 2018; Caswell et al., 2019).
We use SacreBLEU throughout (Post, 2018).
3. Part-of-Speech Tagging
We use part-of-speech taggers for some experiments: universal dependencies (UD) implemented in spaCy 3 and spaCy’s language-specific finegrained POS tags for German from the TIGER Corpus (Albert et al., 2003; Brants et al., 2004).
3. Models
Our unsupervised MT model is a MASS transformer with the hyperparameters of Song et al.
We train MASS on the News Crawl corpora, hereafter called “Unsup.”
Our supervised
MT systems use the transformer-big (Vaswani et al., 2017) as implemented in Lingvo (Shen et al., 2019) with a vocabulary of 32k subword units.
To investigate differences between approaches, we train two language models (LMs) on different types of data and calculate the perplexity of translations generated by the supervised and unsupervised MT systems.
We train one LM on the monolingual German News Crawl dataset with a decoder-only transformer, hereafter called the “natural text LM” (nLM).
We train another on machine translated sentences which we call the “translated text LM” (tLM).
We generate the training corpus by translating the English News Crawl dataset into
German with a English → German transformer-big model trained on the WMT18 bitext.
3. Human Evaluations
Human evaluation complements automatic evaluation and abstracts away from comparison to a human reference which favors the characteristics of translated text (Freitag et al., 2020).
We score adequacy using direct assessment and run side-byside evaluations measuring fluency and adequacy preference between systems.
Each campaign has 1,000 test items.
For side-by-side eval, a test item includes a pair of translations of the same source sentence: one from the supervised system and one from the unsupervised.
We hire 12 professional translators, who are more reliable than crowd workers (Toral, 2020; Freitag et al., 2021).
Direct Assessment
Adequacy
We use the template from the WMT 2019 evaluation campaign.
Human translators assess a translation by how adequately it expresses the meaning of the source sentence on a 0-100 scale.
Unlike WMT, we report the average rating and do not normalize the scores.
Side-by-side Adequacy
Raters see a source sentence with two translations (one supervised, one unsupervised) and rate each on a 6-point scale.
Side-by-side Fluency
Raters assess the alternative translations (one supervised, one unsupervised) without the source, and rate each on a 6-point scale.
Unsupervised vs. Supervised MT
The goal of this section is to analyse supervised and unsupervised systems of similar overall translation quality so that differences in quality do not confound analyses.
As unsupervised systems underperform supervised systems, we use a smaller parallel corpus (news commentary) to train systems of similar quality.
Table 1 summarizes the BLEU scores and human side-by-side adequacy results for both systems.
Although the supervised system is below state-of-the-art, these experiments help elucidate how unsupervised and supervised output is different.
Overall BLEU and human ratings suggest similar translation quality.
Nevertheless, we observe notable differences between orig-de and orig-en sides of the test set when comparing both systems.
Recall that orig-de has natural German text on the target side.
Unsup scores higher than Sup on orig-de, suggesting that its output is more natural-sounding as it better matches text originally written in German.
Performance discrepancies on orig-en and orig-de indicate that differences in system output may exist and prompt further investigation.
[Table]
4. Selecting Translations of Same Adequacy
To assess the translation style and compare linguistic aspects of supervised and unsupervised MT, we further must compare translations that have the same accuracy level on the segment level, so that neither confounds analysis.
We use the adequacy evaluation from Table 1 and retain sentences for which both approaches yield similar adequacy scores.
We divide the rating scale into bins of low (0–2), medium (3–4), and high (5–6) adequacy.
Table 2 shows the percentage of sentences in each bin.
For each source sentence, there is one translation by
Unsup and one by Sup.
If human judges assert that both translations belong in the same adequacy bin, that sentence also appears in “Both.”
There are 86, 255, and 218 sentences in “Both” for low, medium, and high bins, respectively.
For subsequent analyses, we examine sentences falling into “Both.”
[Table]
4. Comparing Translation Style Measuring Structural Similarity
We develop a metric to ascertain the degree of structural similarity between two sentences, regardless of language.
When evaluated on a source-translation pair, it measures the influence of the source structure on the structure of the output without penalizing for differing word choice; thus it is a measure of ”monotonicity” – the degree to which words are translated in-order.
Given alternative translations in the same language, it assesses the degree of structural similarity between the two.
Thus given a machine translation and a human translation of the same source sentence, it can measure the structural similarity between the machine and human translations.
Word alignment seems well-suited here.
Like
Birch (2011), we calculate Kendall’s tau (Kendall, 1938) over alignments of source-translation pairs, but do not simplify alignments to permutations.
We use fast_align (Dyer et al., 2013) but observe that it struggles to align words not on the diagonal, so sometimes skipped alignments.
4 Because of this issue, we instead estimate monotonicity/structural similarity using the new metric, introduced next.
We propose measuring translation edit rate (TER,
Snover et al.
over POS tag sequences .
TER is a well-known word-level translation quality metric which measures the number of edits required to transform a “hypothesis” sentence into the reference, outputting a “rate” by normalizing by sentence length.
Between languages, we compute
TER between POS tag sequences of the source text (considered the reference) and the translation (considered the hypothesis), so that TER now measures changes in structure independent of word choice.
Source-target POS sequences which can be mapped onto each other with few edits are considered similar—a sign of a monotonic translation.
Given a machine translation (hypothesis) and a human reference in the same language, TER over
POS tags measures structural similarity between the machine and human translations .
Outputs with identical POS patterns score 0, increasing to 1+ as sequences diverge.
Lower TER for (source, translation) pairs indicates monotonic translation; Lower
TER for (machine translation, human translation) pairs indicates structural similarity to human translation.
We call the metric “posTER”.
Monotonicity
POS sequences are comparable across languages thanks to universal POS tags.
Table 3 has a toy example with two possible German translations of an English source.
Next to each sentence is its universal dependencies POS sequence.
In the third column, TER is calculated with the
POS sequence of the English source as reference and the sequence of the translation as hypothesis.
Table 4 shows
posTER over universal dependencies of German translations versus the newstest2018 (orig-en) source sentences.
While the standard newstest2018 references (Ref) score 0.410, newstest2018p’s (RefP) higher score of 0.546 reflects the fact that the paraphrase references are designed to have different structure than the source.
Difference in overall monotonicity between
Sup and Unsup is unapparent at this granularity.
Because universal dependencies are designed to
[Table]
POS Sequence
TER
I made myself a cup of coffee this morning.
PRON VERB PRON DET NOUN ADPPNOUN DET NOUN PUNCT
Ich habe mir heute Morgen eine Tasse
PRON AUX PRON ADV NOUN DET 0.
Kaffee gemacht.
NOUN NOUN
VERB PUNCT
Heute morgen habe ich mir eine Tasse
ADV ADV AUX PRON PRON DET 0.
Kaffee gemacht.
NOUN NOUN
VERB PUNCT
[Table]
[Table]
[Table]
suit many languages, the 17 UD categories may be too broad to adequately distinguish moderate structural difference.
Whereas UD has a single class for “VERB,” the finer-grained German TIGER tags distinguish between 8 sub-verb types including infinitive, modal, and imperative.
We use these languagespecific categories next to uncover differences between systems that broad categories conceal.
Similarity to Human Translation
Recall that supervised MT essentially mimics human translators, while unsupervised MT learns to translate without examples.
Intuitively, supervised MT output might be stylistically more like human translation, even when controlling for quality.
The first indication is
Sup’s lower BLEU score on nt18p—the paraphrase test set designed to have structure different than the original human translation.
We compare the structure of MT output with the human reference using German TIGER tags.
Lower posTER indicates more structural similarity, while higher posTER indicates stylistic deviation from human translation.
Comparison with the newstest2018 orig-en human reference is in Table 5.
Sup and Unsup show negligible difference overall, but binning by adequacy shows Unsup output as less structurally similar to the human reference on the high-end of adequacy , and more similar on the low-end.
This suggests systematic difference between systems, and that unsupervised MT might have more structural diversity as quality improves.
[Table]
Naturalness
The first hint that Unsup might produce more natural output than Sup is its markedly higher BLEU on the orig-de test set: 27.1, versus 21.1 from Sup.
Recall that orig-de has natural German on the target side, so higher BLEU here means higher n-gram overlap with natural German.
Edunov et al.
(2020) recommend augmenting
BLEU-based evaluation with perplexity from a language model (LM) to assess fluency or naturalness of MT output.
Perplexity (Jelinek et al., 1977) measures similarity of a text sample to a model’s training data.
We contrast the likelihood of output according to two LMs: one trained on machinetranslated text (tLM) and another trained on nontranslated natural text (nLM).
While machinetranslated and human-translated text differ,
the
LMs are nonetheless a valuable heuristic and contribute insights on whether systematic differences between MT system outputs exist.
Low perplexity from the nLM indicates natural language.
Low perplexity from the tLM (trained on English News
Crawl that has been machine-translated into German) shows proximity to training data composed of translated text, indicating simplified language.
Sup perplexity is lower than Unsup across adequacy bins for the tLM, seen in Table 6.
Conversely,
Sup generally has higher perplexity from the nLM.
All adequacy levels for Unsup have similar nLM perplexity, suggesting it is particularly skilled at generating fluent output.
Together, these findings suggest that unsupervised MT output is more natural than supervised MT output.
Stronger Supervised MT
Though analyzing systems of similar quality is important for head-tohead comparison, we evaluate a stronger supervised system for context.
5 We do not have human evaluation scores, but automatic results give insight: see Table 7.
The model has overall BLEU =
40.9 and a similarly large discrepancy on orig-en vs. orig-de as did the Sup system used throughout this
Natural Text
[Table]
work: 44.6 for orig-en and 34.9 for orig-de.
As for structural similarity, this stronger system has lower overall posTER vs. the human reference—0.
vs. 0.280/0.287 from Sup/Unsup—indicating even more structural similarity with the reference.
For naturalness, the stronger system has lower perplexity from the nLM.
As a higher-quality system, this is expected.
At the same time, it scores much lower than Sup and Unsup by the tLM, where higher indicates more natural-sounding output: 29.23 vs. 41.06/58.17 for Sup/Unsup.
[Table]
Structural sim.
is posTER: v. Src is comparable to Table 4, v
Ref to nLM/tLM are Natural/Translated Text LMs of Table 6.
Ablation: Architecture vs. Data One
reason
Unsup might produce more natural-sounding output could be simply that it develops languagemodeling capabilities from natural German alone, whereas Sup must see some translated data (being trained on bitext of human translations).
Next, we ask whether the improved naturalness and structural diversity is due to the unsupervised NMT architecture, or simply the natural training data.
We build a supervised en-de MT system with 329,000 paired lines of translated English source and natural German, where the source is backtranslated German News Crawl from a supervised system.
In other words, we train on backtranslated data only on the source side and natural German as the target.
The model thus develops its languagemodeling capabilities on natural sentences alone.
If more natural output is simply a response to training on natural data, then this supervised system should perform as well in naturalness as Unsup, or better.
We train another unsupervised system on translated text only.
Source-side training data is synthetic English from translating German News
Crawl with a supervised system.
Target-side is synthetic German which was machine-translated from
English News Crawl.
If naturalness solely results from data, this system should perform worst, being trained only on translated (unnatural) text.
Table 8 shows the results.
The original unsupervised system (Unsup) performs best according to both LMs, having output that is more natural and less like translated text.
When given only natural
German to build a language model, the supervised system (Sup En-Trns/De-Orig) still produces more unnatural output than Unsup.
Even when the unsupervised system uses translated data only (UnsupTrns), its output is still more natural than the original supervised system (Sup) according to both LMs.
This is a surprising result, and is interesting for future study.
Together, these findings suggest that both German-original data and the unsupervised architecture encourage output to sound more natural.
Application: Leveraging Unsupervised Back-translation
Our results indicate that high-adequacy unsupervised MT output is more natural and more structurally diverse in comparison to human translation, than is supervised MT output.
We are thus motivated to use these advantages to improve translation.
We explore how to incorporate unsupervised
MT into a supervised system via back-translation.
We train for ∼ 500,000 updates for each experiment, and select models based on validation performance on newstest2018.
We test on newstest2019(p).
5. Baselines
The first row of Table 9 is the supervised baseline trained on the WMT18 bitext.
The second row is
Unsup, used throughout this work.
We back-translate 24 million randomly-selected sentences of German News Crawl twice: once using a supervised German-English system trained on WMT18 bitext with a transformer-big architecture, and once using Unsup.
Both use greedy decoding for efficiency.
We augment the WMT18 bitext with either the supervised or unsupervised BT.
Seen in Table 9, adding supervised BT (+SupBT) performs as expected; minorly declining on the source-original test set (orig-en), improving on the target-original set (orig-de), and improving on
[Table]
the paraphrase set (nt19p).
Conversely, adding unsupervised BT (+UnsupBT) severely lowers
BLEU on source-original and paraphrase test sets.
Randomly-partitioning the BT sentences such that 50% are supervised BT and 50% are unsupervised also lowers performance on orig-en (+50-50BT).
5. Tagged BT
Following Caswell et al.
(2019), we tag BT on the source-side.
Tagging aids supervised BT (+SupBT_Tag) and greatly improves unsupervised
BT (+UnsupBT_Tag), which outperforms the baseline and is nearly on-par with +SupBT_Tag.
Combining supervised and unsupervised BT using the same tag for both (+50-50BT_Tag) shows no benefit over +SupBT_Tag.
+50-50BT_TagDiff uses different tags for supervised vs. unsupervised BT.
5. Probability-Based BT Selection
We design a BT selection method based on translation probability to exclude unsupervised BT of low quality.
We assume that supervised BT is “good enough.”
Given translations of the same source sentence (one supervised, one unsupervised) we assert that an unsupervised translation is “good enough” if its translation probability is similar or better than that of the supervised translation.
If much lower, the unsupervised output may be low-quality.
• Score each supervised and unsupervised BT with a supervised de-en system.
• Normalize the translation probabilities to control for translation difficulty and output length.
• Compare probability of the supervised and unsupervised BT of the same source sentence:
[Equation]
• Sort translation pairs by ∆ P. • Select the unsupervised BT for pairs scoring highest ∆ P and the supervised BT for the rest.
This filters out unsupervised outputs less than a hyperparameter T% as likely as the corresponding supervised sentence and swaps them with the corresponding supervised sentence.
Importantly, the same 24M source sentences are used in all experiments.
The procedure is shown in Figure 1.
Full results varying T are in the Appendix for brevity, but we show two example systems in Table 9.
The model we call “+MediumMix_Tag” uses the top ∼ 40% of ranked unsupervised BT with the rest supervised (9.4M unsupervised, 14.6M supervised).
“+SmallMix_Tag” uses the top ∼ 13% of unsupervised BT (3.1M unsupervised, 20.9M supervised).
6 We use the same tag for all BTs.
Improvements are modest, but our goal was to demonstrate how one might use unsupervised MT output rather than build a state-of-the-art system.
+SmallMix_Tag performs better than the previous best on newstest2018p and +MediumMix_Tag performs highest overall on nt19p.
We recall
[Table]
that small differences on paraphrase test sets can signal tangible quality differences (Freitag et al., 2020).
Trusting BLEU on nt19p, we use +MediumMix_Tag as our model for human evaluation.
One might inquire whether improved performance is due to the simple addition of noise in light of Edunov et al.
(2018), who conclude that noising
BT improves MT quality.
Subsequent work, however, found that benefit is not from the noise itself but rather that noise helps the system distinguish between parallel and synthetic data (Caswell et al., 2019; Marie et al., 2020).
Yang et al.
(2019) also propose tagging to distinguish synthetic data.
With tagging instead of noising, Caswell et al.
(2019) outperform Edunov et al.
(2018) in 4 of 6 test sets for En-De, furthermore find that noising on top of tagging does not help.
They conclude that “tagging and noising are not orthogonal signals but rather different means to the same end.”
In light of this, our improved results are likely not due to increased noise but rather to systematic differences between supervised and unsupervised BT.
5. Human Evaluation
We run human evaluation with professional translators for +MediumMix_Tag , comparing its output translation of the newstest2019 test set with two baseline models.
Table 10 shows that humans prefer the combined system over the baseline outputs.
7 Table 11 shows the percentage of sentences judged as “worse than,” “about the same as,” or “better than” the corresponding +SupBT_Tag output, based on fluency.
Raters again prefer the combined system.
The improvements are modest, but
Adequacy + UnsupBT_Tag
[Table]
[Table]
encouragingly indicate that unsupervised MT may have something to contribute to machine translation, even in high-resource settings.
Conclusion
Recent unsupervised MT systems can reach reasonable translation quality under clean and controlled data conditions, and could bring alternative translations to language pairs with ample parallel data.
We perform the first systematic comparison of supervised and unsupervised MT output from systems of similar quality.
We find that systematic differences do exist, and that high-quality unsupervised
MT output appears more natural and more structurally diverse when compared to human translation, than does supervised MT output.
Our findings indicate that there may be useful differences between supervised and unsupervised MT systems that could contribute to a system better than either achieves alone.
As a first step, we demonstrate an unsupervised back-translation augmented model that takes advantage of the differences between the translation methodologies to outperform a traditional supervised system on human-evaluated measures of adequacy and fluency.