Part Represents Whole: Improving the Evaluation of Machine Translation System Using Entropy Enhanced Metrics (2022)

Abstract: Machine translation (MT) metrics often experience poor correlations with human assessments. In terms of MT system evaluation, most metrics pay equal attentions to every sample in an evaluation set, while in human evaluation, difficult sentences often make candidate systems distinguishable via notable fluctuations in human scores, especially when systems are competitive. We find that samples with high entropy values, which though usually count less than 5%, tend to play a key role in MT evaluation: when the evaluation set is shrunk to only the high-entropy portion, correlations with human assessments are actually improved. Thus, in this paper, we propose a fast and unsupervised approach to enhance MT metrics using entropy, expanding the dimension of evaluation by introducing sentence-level difficulty. A translation hypothesis with a significantly high entropy value is considered difficult and receives a large weight in aggregation of system-level scores. Experimental results on five sub-tracks in the WMT19 Metrics shared tasks show that our proposed method significantly enhanced the performance of commonly-used MT metrics in terms of system-level correlations with human assessments, even outperforming existing SOTA metrics. In particular, all enhanced metrics exhibit overall stability in correlations with human assessments in circumstances where only competitive MT systems are included, while the corresponding vanilla metrics fail to correlate with human assessments.

## introduction automatic evaluation plays an indispensable role in the evaluation of machine translation (mt) systems, working as a proxy of human assessment as well as a promising approach to give instant feedback during the development of mt systems. however, it has been a challenge for automatic evaluations to correlate with human judgement. for instance, major discrepancy is detected between human assessments and automatic evaluations in terms of system ranking in wmt19 english-german evaluation tasks @xcite . experiments conducted by @xcite and @xcite further indicate that when inferior systems are excluded, current automatic metrics expect major falling on correlations with human referees, sometimes even down to the degree of negative correlations. in order to improve the evaluation of mt systems, many meticulously designed metrics are proposed. however, popular mt metrics focus on a segment-level comparison between references and hypotheses, and output system-level scores by a simple arithmetic average over segment scores, ignoring the differences among samples in an evaluation set @xcite @xcite . in contrast, the core idea of assigning different weights to samples in a dataset is proven effective in the field of curriculum learning @xcite . for mt evaluation, it is not likely that human raters treat every source-reference pair equally. those simple samples can be easily translated, leading to similar human scores given to different hypotheses, while the more challenging part in an evaluation set often distinguishes top candidates from inferior systems. inspired by recent work of @xcite , who determine the difficulty of sub-units in translation hypotheses by reviewing performances of corresponding sub-units among k candidate systems, we further introduce sentence-level difficulty into mt evaluation, which functions as a weight in the aggregation of final system scores. in determination of proposed sentence-level difficulty, instead of using an embedding-based approach similar to zhan et al.'s, we adopt a fast and unsupervised entropy-based measurement. in information theory, entropy is a measure of the uncertainty in a random variable. the entropy h of a discrete random variable x with possible values x 1 , x 2 , ..., x n is defined by @xcite as it is surprising to see a straightforward implementation under the idea of sentence-level difficulty weights based on entropy, involving no deeplearning techniques, yet enhanced the performance of a bert-based mt metric. the aim of this paper is to introduce the concepts and show the effective roles entropy and sentence-level difficulty play in enhancing mt evaluation quality, but not to explore optimal techniques integrating them into mt evaluation. ## related work existing reference-based mt metrics can be roughly categorized into three types: matchingbased metrics @xcite @xcite @xcite , embedding-based metrics @xcite @xcite and end-to-end metrics @xcite . matching-based metrics estimate quality of translation by hand-crafted features, such as n-grams, edit distance and alignments. bleu @xcite is a classical criterion based on word-level n-gram matching between references and hypothesis and is widely employed as baselines in mt system evaluation, while chrf @xcite computes an f-score based on character-level ngrams. meteor @xcite focuses on semantic matched chunks acquired by alignment, where lengths of chunks are dynamically determined and the limitation of maximum matching length of n-gram based metrics is partially relieved. in contrast, bertscore and its variants @xcite , owing to powerful contextual embedding acquired from modern language models, catch deep-level semantic information inside the translation pairs and achieve high rankings across mt evaluation benchmarks in terms of correlations with human assessments. ## conclusion and future work in this paper, we find that the high-entropy hypotheses, though holding only a minor portion in an evaluation set, play a significant role in terms of correlations with human judgments in mt evaluation. by rebalancing the weights between lowentropy and high-entropy hypotheses, an entropy enhancing approach for mt metrics is proposed. experimental results on five sub-tracks in wmt19 metric tasks show that our proposed approach successfully enhances the performance of popular mt metrics and achieves remarkable correlations with human assessments, especially in the evaluation of competitive systems. our analysis introduces the concept of sentence-level difficulty into mt evaluation and reveals the importance of difficult samples in system-level evaluations. there are several directions for future exploration. first, entropy-based difficulty can work as a measurement to the quality of an mt evaluation set. if an evaluation set contains more high-entropy samples, its ability to rank systems is better. second, using entropy, we can dig the hard samples out of an evaluation set and, by filtering easy samples, we can make a distillation of evaluation set. third, there is still room for optimization in calculation of entropy and difficulty weights.