Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation (2024)

Abstract: A good translation should be faithful to the source and should respect the norms of the target language. We address a theoretical puzzle about the relationship between these objectives. On one hand, intuition and some prior work suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source can only be achieved at the cost of fluency. On the other hand, quality assessment researchers often suggest that accuracy and fluency are highly correlated and difficult for human raters to distinguish (Callison-Burch et al., 2007). We show that the tension between these views is an instance of Simpson’s paradox, and that accuracy and fluency are positively correlated at the level of the corpus but trade off at the level of individual source segments. We further suggest that the relationship between accuracy and fluency is best evaluated at the segment (or sentence) level, and that the trade off between these dimensions has implications both for assessing translation quality and developing improved MT systems.

## introduction no translation can simultaneously satisfy all possible goals, and translation is therefore an art of navigating competing objectives @xcite . many objectives are discussed in the literature, but two in particular seem especially fundamental. the first is accuracy (also known as fidelity or adequacy), or the goal of preserving the information in the source text (st). the second is fluency, or the goal of producing target text (tt) that respects the norms of the target language (tl) and is easy for the recipient to process @xcite . here we study the relationship between accuracy and fluency and work with two operationalizations of these notions. the first relies on human judgments of accuracy and fluency collected in prior work on translation quality estimation @xcite . the second relies on probabilities estimated using neural machine translation (nmt) models. given a source-translation pair (x, y), p(x|y) corresponds to accuracy, and p(y) corresponds to fluency @xcite . p(x|y) will be low if y fails to preserve all of the information in x, and p(y) will be low if y violates the norms of the target language. to highlight that model estimates p(x|y) and p(y) are related to but distinct from human ratings of accuracy and fluency, we refer to p(x|y) as accuracy m and p(y) as fluency m . some parts of the literature argue that accuracy trades off with fluency. in figure 1a , the blue dots are translations of the same source segment, and an opposing view of the relationship between accuracy and fluency emerges from the literature on quality estimation. here the common wisdom is that accuracy and fluency are highly correlated and practically indistinguishable to human annotators @xcite @xcite @xcite ). as a result, accuracy and fluency are conflated as a single assessment score in recent wmt general machine translation tasks, with more emphasis given to accuracy than fluency @xcite 2021; @xcite . we argue that the conflict between these views is an instance of simpson's paradox @xcite , which occurs when a relationship at one level of analysis (e.g. the corpus level) disappears or is reversed at a different level (e.g. the segment or sentence level). figure 1 shows how the correlation r c between accuracy and fluency can be positive over a miniature corpus including translations of three source segments even though the correlation r s for each individual source segment is negative. of the two levels of analysis, the segment level is the appropriate level for understanding how humans and machine translation systems should choose among possible translations of a source segment. the central goal of our work is therefore to establish that the correlation between accuracy and fluency is negative at the level of individual source segments. ## tradeoff between p(x|y) and p(y) because accuracy m and fluency m have formal definitions, we start with these dimensions. ## tradeoff between accuracy and fluency we now turn to human ratings of accuracy and fluency, and demonstrate that the two are again negatively correlated at the segment level. data. only rltc and mtmqm are rated by human annotators. the subset of rltc released by @xcite includes accuracy and fluency scores derived from error annotations. for mtmqm, we follow @xcite where accuracy scores are aggregates of "accuracy" and "terminology" errors, and fluency scores are aggregates of "fluency", "style" and "locale convention" errors. targets that are labelled "nontranslation" receive scores of zero for both accuracy and fluency. major and minor errors receive penalties of 5 and 1 respectively. fluency/punctuation is assigned a penalty of 0.1. we calculate the final rating as s @xmath0, 25e c ), where e c denotes the total penalty in error category c. 6 because some systems submit the same translation but receive different ratings, we average these scores and remove the duplicate entries. results. figure 4 shows correlations at the level of individual source segments. the majority of correlations are negative, and paired-sample t-tests reveal that both distributions are significantly (@xmath1.001) different from distributions obtained from random permutations. the results therefore suggest that accuracy and fluency (as rated by humans) trade off at the level of individual segments. at the corpus level, accuracy and fluency are positively correlated for mtmqm (r @xmath2.392, @xmath3.001), and are uncorrelated in rltc (r @xmath4.085, @xmath5.001), suggesting again that simpson's paradox applies to both cases. 7 unlike the case for accuracy m and fluency m , human ratings of accuracy and fluency do not induce a positive correlation between p(x) and r s (@xmath6.150 and -.104 for rltc and mtmqm respectively). we therefore find no support for the simulation-based prediction that low-probability sentences are more likely to produce strong tradeoffs between accuracy and fluency. figure 4 is directly analogous to figure 3 , and we expected that source segments which showed strong tradeoffs (i.e. extreme negative correlations) in figure 3 would also show strong tradeoffs in figure 4 . the two tradeoff measures, however, were uncorrelated, 8 which suggests that accuracy m and fluency m overlap only partially with human ratings of accuracy and fluency. a similar conclusion is suggested by figure 5 , which shows pearson correlations of translation probability (p(y|x); blue bars), accuracy m (p(x|y); brown bars) and fluency m (p(y); green bars) with human ratings of accuracy and fluency for rltc and mtmqm. 9 as expected, accuracy m shows a higher correlation with accuracy than fluency, and fluency m shows the opposite pattern. figure 5 however, suggests that accuracy m is not superior to p(y|x) as a predictor of accuracy, and that fluency m is not superior to p(y|x) as a predictor of fluency. one reason why our model estimates of accuracy and fluency depart from human ratings is that accuracy m and fluency m are sensitive to segment length. for example, a longer segment will have lower fluency m than a shorter segment even if the two are both perfectly idiomatic. ## conclusion we showed that accuracy and fluency and p(x|y) and p(y) both trade off when translating individual source segments. this finding suggests that current protocols for assessing translation quality may need to be adjusted. human assessments for recent wmt general tasks are performed using direct assessment and scalar quality metrics (da+sqm) @xcite . this approach conflates meaning preservation and grammar into a single score indicative of overall quality of a transcorpus level because of a ceiling effect -63.5% and 70.6% of sentences receive maximum ratings for fluency and accuracy in rltc compared to 55.6% and 58.4% for mtmqm. 8 the pearson correlations between the two tradeoff measures for rltc and mtmqm are @xmath7.003, @xmath8.933 and @xmath9.022, @xmath10.05. 9 values are in log scale and are ranked by percentile. acc. flu. zhen and ende refer to chinese-english and english-german language pairs. all correlations reported are significant (@xmath11.001). our results also suggest the value of developing mt models that navigate the accuracy-fluency tradeoff in human-like ways. in some settings (e.g. translating legal texts) accuracy is more important than fluency @xcite @xcite @xcite , but in others (e.g. translating informal conversation) fluency may take priority @xcite . one natural approach to navigating the accuracy-fluency tradeoff builds on noisy channel models @xcite @xcite , which incorporate both p(y) and p(x|y) along with tradeoff parameters that specify the relative weights of the two. tuning these parameters for specific registers may allow a model to find the right balance between accuracy and fluency in each case.