Feeding Two Birds or Favoring One? Adequacy–Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation (2025)

Abstract: We investigate the tradeoff between adequacy and fluency in machine translation. We show the severity of this tradeoff at the evaluation level and analyze where popular metrics fall within it. Essentially, current metrics generally lean toward adequacy, meaning that their scores correlate more strongly with the adequacy of translations than with fluency. More importantly, we find that this tradeoff also persists at the meta-evaluation level, and that the standard WMT meta-evaluation favors adequacy-oriented metrics over fluency-oriented ones. We show that this bias is partially attributed to the composition of the systems included in the meta-evaluation datasets. To control this bias, we propose a method that synthesizes translation systems in meta-evaluation. Our findings highlight the importance of understanding this tradeoff in meta-evaluation and its impact on metric rankings.

## introduction as translation systems become more sophisticated and widely adopted, the critical challenge of accurately evaluating their performance has come to the forefront, driving significant ongoing research to the developement of more accurate and robust metrics @xcite . traditionally, translation evaluation relied on lexical metrics such as bleu @xcite and chrf @xcite , which primarily consider the n-gram overlap between the candidate and reference translations. however, these metrics have been shown insufficient for measuring the quality of modern translation systems @xcite . in the recent decade, researchers have been exploring the idea of training neural models to measure translation quality @xcite @xcite , with popular ⋆ work partially done when the first author was interning at google. examples including metricx @xcite and comet @xcite . two key aspects of translation quality, long discussed in the community @xcite @xcite , are foot_0 : the aforementioned tradeoff naturally results in an adequacy-fluency tradeoff at the level of evaluation metrics, if we limit our evaluation to a single score: increasing the capability of measuring one aspect eventually comes at the cost of decreasing the capability of measuring the other. it is important to understand this tradeoff and know the position of each metric to avoid undesired biases when optimizing translation systems according to the metric. in this work, we demonstrate the severity of the adequacy-fluency tradeoff at the evaluation level. we analyze current evaluation setups in wmt @xcite , and show that there is a significant disagreement between adequacy and fluency when ranking systems ( ## analysis: tradeoff at the meta-evaluation level in this section, we explore the adequacy-fluency tradeoff at the meta-evaluation level. we first discuss in §4.1 how the tradeoff exists in meta-evaluation, and show that the imbalance in wmt meta-evaluation datasets imposes a bias that favors adequacy-oriented metrics. we argue that this bias consists of both an intrinsic and an extrinsic component, and that we aim to control the latter. in §4.2, we design a metric to quantify the extrinsic bias, and in §4.3, we propose a practical approach to reduce it. finally we demonstrate in §4.4 the impact of meta-evaluation bias on the ranking of translation metrics by comparing this ranking before and after reducing the meta-evaluation extrinsic bias. ## analysis: tradeoff at the evaluation level in this section, we analyze the adequacy-fluency tradeoff at the evaluation level and determine the bias of each metric within this tradeoff. note that the notion of bias considered here is related to but distinct from that in §4. we say a metric is biased toward adequacy or fluency if it ranks translation systems in a way that is more aligned with the ranking derived solely from that aspect, as measured by the corresponding mqm scores. to study the aforementioned bias, we would like to separately meta-evaluate the adequacy and fluency of each metric, using the original wmt metaevaluation setup and our balanced setup described in row 6 of from the data statistics in despite the significant discordance between adequacy mqm and fluency mqm, we also observe a large number of concordant cases (i.e., one system outperforms another in both adequacy mqm and fluency mqm). this raises a challenge when we separately analyze adequacy and fluency. to address this, we design three experimental protocols, as follows. ## conclusion in this work, we investigate the adequacy-fluency tradeoff in translation. while this tension is well-documented at the level of translation output @xcite , we show that it also manifests severely at the levels of evaluation and meta-evaluation. through empirical analysis of wmt metaevaluation protocols, we uncover a systematic bias toward adequacy, driven by the composition of meta-evaluation datasets. we also propose a method that can control the balance between adequacy and fluency. we further analyze the placement of popular translation evaluation metrics along the adequacy-fluency tradeoff and find that most metrics lean toward adequacy. we argue that the adequacy-fluency tradeoff is a critical yet under-recognized matter in translation evaluation and meta-evaluation. while we do not take a stance on how to deal with such bias, our primary contribution is to raise awareness of this matter within the community. we discuss future work in appendix g.