To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translation (2024)

Abstract: We conduct a large-scale fine-grained comparative analysis of machine translations (MTs) against human translations (HTs) through the lens of morphosyntactic divergence. Across three language pairs and two types of divergence defined as the structural difference between the source and the target, MT is consistently more conservative than HT, with less morphosyntactic diversity, more convergent patterns, and more one-to-one alignments. Through analysis on different decoding algorithms, we attribute this discrepancy to the use of beam search that biases MT towards more convergent patterns. This bias is most amplified when the convergent pattern appears around 50% of the time in training data. Lastly, we show that for a majority of morphosyntactic divergences, their presence in HT is correlated with decreased MT performance, presenting a greater challenge for MT systems.

## introduction translation divergences occur when the translations differ structurally from the source sentences, typically as a result of either inherent crosslingual differences or idiosyncratic preferences of translators. these divergences happen naturally in the translation process and can be readily found in human translations (hts), including those used for training machine translation (mt) systems (see the table in figure 1 for some examples). their existence in ht has long been regarded as a key challenge for mt @xcite and more recent empirical studies have demonstrated the abundance of translation divergences in ht @xcite . in contrast to ht, mt outputs tend to be less diverse and more literal (i.e., absence of translation divergence), exhibiting the features of translationese @xcite . this qualitative difference between ht and mt has inspired a rich body of work attempting to narrow the gap, such as automatic detection of machine translated texts in the training data @xcite @xcite @xcite , training mt systems on more diverse translations @xcite , and carefully reordering the examples to reduce the degree of divergence between the source and the target @xcite @xcite . the challenges that translation divergences present do not just concern training mt systems, but also their evaluation @xcite . nonetheless, even as we gain deepened understanding of how to address these challenges, it remains unclear how quantitatively different mt and ht are in terms of divergences. 1 control verbs, 2 for instance, provide a great case study to showcase this difference. there is much uncertainty when translating them from english to french, and human translators employ a wide variety of constructions including many divergent patterns (figure 1 ). in comparison, mt is much more likely to preserve the source structure, with the convergent pattern constituting about 20% more of all translations of control verbs. this difference exemplifies mt's undesirable tendency to produce translationese that is too literal and lacks structural diversity @xcite . in this work, we seek to systematically investigate this difference by conducting a large-scale fine-grained comparative analysis on the distribution of translation divergences for ht and mt, all : examples of divergences in ht for @xmath0, with relevant fragments of the source/target shown in the first/second rows. the english control constructions are bolded including both the finite root verb and the controlled word, while the french phrases of interest are underlined. bottom figure: percentages of target patterns for ht and mt, with obligatory control finite verbs as the source pattern. o2o:conv: one-to-one convergent patterns where the target phrase uses a similar control construction to the source; o2o:div: one-to-one divergent patterns where the target differs structurally from the source; null: no target word is aligned; others: other less frequent patterns (e.g., one-to-many alignments). the percentages of all four categories sum up to 100%. through extensive analyses based on three language pairs and two types of morphosyntactic divergence using the annotational framework of universal dependencies @xcite , we make the following empirical observations: to the best of our knowledge, this is the first work to present the comparative perspective of ht vs mt in such fine granularity covering thousands of morphosyntactic constructions. in the remaining sections, we first briefly describe related work in section 2. the experimental setup is described in detail in section 3. we demonstrate the quantitative difference between mt and ht in section 4, and seek to understand this discrepancy in section 5. lastly, we explore the correlation between the presence of divergences in ht with mt performance in section 6 and make conclusions in section 7. the closest work to ours is from @xcite , who proposed to investigate fine-grained crosslingual morphosyntactic divergence based on universal dependencies. they augmented a subset of the parallel universal dependencies (pud) corpus @xcite with human-annotated word alignments for five language pairs and focused exclusively on content words. while our work shares a similar conceptional and methodological foundation to theirs, our goal is to conduct a comparative analysis between ht and mt. in addition, we rely on a dependency parser and a word aligner (see section 3 for more details) to reach a sufficiently large scale to enable the investigation of more fine-grained divergences. diverse machine translation mt systems tend to produce less diverse outputs in general @xcite , which is particularly harmful for back translation @xcite @xcite . to address this issue, various techniques have been proposed in the literature, including modified decoding algorithms @xcite @xcite , mixtures of experts @xcite , bayesian models @xcite , additional codes (syntax or latent) @xcite , and training with simulated multireference corpora @xcite . in all aforementioned works, the emphasis is on the lack of diversity in mt outputs rather than comparing them systematically against ht. notable exceptions include @xcite , who investigated the distributional differences between mt and ht in terms of n-grams, sentence length, punctuation, and copy rates. @xcite compared translations from supervised mt and unsupervised mt and noted their systematic style differences based on similarity and monotonicity in their pos sequences. in contrast, our work goes beyond surface features and focuses on fine-grained morphosyntactic divergences. algorithmic bias another closely related line of work studies algorithmic biases of current nlp systems, with particular emphasis on gender and racial biases @xcite @xcite . specifically for mt, researchers have focused on lexical diversity by comparing ht against post-editese @xcite or mt outputs directly @xcite ; @xcite have compared ht, mt, and simultaneous interpreting in terms of translationese using pos perplexity and dependency length. most related to our work, @xcite have conducted an extensive comparison between ht and mt based on a suite of lexical and morphological diversity metrics. while our study reaches a similar conclusion that mt is less diverse than ht, we explore morphosyntactic patterns on a more fine-grained level, and also reveal the bias of mt (and more specifically beam search) towards convergent structures. ## experimental setup types of morphosyntactic divergence in this study, we experiment with two types of translation patterns based on the annotational scheme of universal dependencies: these types are largely based on the proposal of @xcite , with modifications to accommodate more granularity. with either type, the translation pattern is a convergence if the source and the target sides have the same structure (word-based or arc-based), and otherwise a divergence. notationally, we use tildes to connect the various parts of the pattern in a fixed order. for instance, for the control verb ''cautioned'' in figure 2 , its word-based divergence has root ˜verb ˜nsubj+xcomp on the source side, where verb corresponds to its pos tag, root its parent dependency, and nsubj and xcomp its two child dependencies. similarly, we have root ˜verb ˜nsubj+obl+xcomp on the data we conduct experiments for three language pairs using wmt datasets @xcite : @xmath1, @xmath2, and @xmath3. all training datasets are lightly filtered based on length, length ratio, and language id, and deduplicated. for each language pair, one million sentences are held out from the training split to form an analysis subset. all analyses in our study are based on this subset to eliminate potential confounding effects from domain mismatch. ## comparative analysis of mt vs ht we proceed to conduct a comparative analysis of mt vs ht based on the fine-grained morphosyntactic patterns defined in the previous section. for any given source pattern p according to the word-based or arc-based definition as detailed in the previous section, we study the distribution of its aligned target patterns, i.e., pr ht (• | p) and pr mt (• | p), along two major dimensions: diversity/uncertainty as measured by entropy of the target pattern, and convergence/divergence rate. figure 3 shows that there is considerable variance in how the most frequent source patterns in ht are distributed along these two axes, and that each dimension captures a different property of the distribution. through analyses on both the aggregate level and the individual pattern level, we conclude that mt is more conservative than ht, with less morphosyntactic diversity, more convergent patterns, and more one-to-one alignments. we also observe that mt tends to be less similar to ht for the less frequent source patterns. the analyses in this section are based on the held-out subset consisting of one million sentence pairs. we refer readers to appendix a for similar results on a subset that is further filtered using labse crosslingual embeddings @xcite with a remarkably similar trend, which we include to show that it does not change our conclusions when we test on data that has been filtered to improve its cross-lingual equivalence. ## understanding the discrepancy in this section, we seek to understand the source of discrepancy between ht and mt as demonstrated in the previous section. by investigating different decoding algorithms, we attribute this discrepancy to the use of beam search, echoing the thesis laid out by previous work @xcite . more specifically in our experiments, we show that beam search is biased towards less diverse and more convergent translations, even when the learned model distribution actually resembles ht. this bias is most prominent when the convergent patterns appear around 50% of the time in training data. moreover, frequencies of convergent patterns in mt are increased even when they are uncommon in ht, suggesting perhaps a more inherent structural bias in current mt architectures. ## divergence and mt quality in our final analysis, we investigate how the presence of morphosyntactic divergence in ht might affect mt quality. in contrast to the previous sections analyzing conditional distributions given a source pattern, we focus instead on individual divergences/convergences. the potential connection between divergence and mt quality is motivated by second-language acquisition research that describes language inference from their first languages (i.e., negative transfer) as one source of difficulty for learners @xcite , which can happen when the two languages diverge structurally. do mt systems have similar problems with divergences? preliminaries to answer this question, we conduct an analysis on the presence (or absence) of a word-based morphosyntactic divergence in ht and the corresponding mt quality as measured by bleu @xcite and bleurt @xcite . the basic idea is to construct two contrastive groups of source sentences (called the experiment group and the control group) and compare the mt performance on each group. the ht references of the experiment group contain a given divergent pattern, corresponding to sentences that are perhaps more challenging 12 we do not observe a similar trend when comparing ancestral sampling against ht. to translate, whereas those of the control group do not. more specifically, for a given divergence with source pattern p and target pattern q (@xmath4, its control group consists of source sentences for which ht translates every source p into target p (i.e., a convergent pattern), and its experiment group consists of source sentences for which ht translates every source p into p except for one that is translated into q. for an simplified example, if we are interested the divergence that translates nouns into verbs, the corresponding control group contains source sentences for which ht translates every noun into a noun, whereas its experiment group contains source sentences for which exactly one noun is translated into a verb and the rest of nouns into nouns. we then collect the mt outputs for both groups and compute the differences in bleu and bleurt. this procedure is repeated for every divergence pattern for which both groups have at least 100 sentences. findings we treat each difference in bleu or bleurt as one data point and plot their estimated probability density function. as illustrated in figure 10 , divergences are more often associated with significantly lower bleu scores (i.e., negative differences), with a fairly large amount of what could explain this variance? why are some divergent patterns associated with worse mt performance while others aren't? one obvious hypothesis is that these patterns are seen less frequently during training. however, a closer inspection seems to suggest that frequency of divergent patterns alone is not an adequate predictor. more specifically, we use the absolute or relative frequency foot_8 of the divergent pattern, with or without taking a log of the number, and correlate it with bleu or bleurt scores. even with the best option (log of relative frequency) presented in we conduct a large-scale fine-grained comparative investigation between ht and mt outputs, through the lens of morphosyntactic divergence. based on extensive analyses on three language pairs, we demonstrate that mt is less morphosyntactic diverse and more convergent than ht. we further attribute to this difference to the use of beam search that biases mt outputs towards less diverse and less divergent patterns. finally, we show that the presence of divergent patterns in ht has overall an adverse effect on mt quality. in future work, we are interested in applying the same analysis to large language model (llm)-based mt systems. recent studies have noted that llm-based systems tend to produce less literal translations, compared to the traditional encoder-decoder models @xcite . it would be interested to see whether and to what extent the llm translations might differ from those produced by traditional models when viewed from a morphological lens.