Difficulty-Aware Machine Translation Evaluation (2021)

Abstract: The high-quality translation results produced by machine translation (MT) systems still pose a huge challenge for automatic evaluation. Current MT evaluation pays the same attention to each sentence component, while the questions of real-world examinations (e.g., university examinations) have different difficulties and weightings. In this paper, we propose a novel difficulty-aware MT evaluation metric, expanding the evaluation dimension by taking translation difficulty into consideration. A translation that fails to be predicted by most MT systems will be treated as a difficult one and assigned a large weight in the final score function, and conversely. Experimental results on the WMT19 English-German Metrics shared tasks show that our proposed method outperforms commonly used MT metrics in terms of human correlation. In particular, our proposed method performs well even when all the MT systems are very competitive, which is when most existing metrics fail to distinguish between them. The source code is freely available at https://github.com/NLP2CT/Difficulty-Aware-MT-Evaluation .

## introduction the human labor needed to evaluate machine translation (mt) evaluation is expensive. to alleviate this, various automatic evaluation metrics are continuously being introduced to correlate with human judgements. unfortunately, cutting-edge mt systems are too close in performance and generation style for such metrics to rank systems. even for a metric whose correlation is reliable in most cases, empirical research has shown that it poorly correlates with human ratings when evaluating competitive systems @xcite , limiting the development of mt systems. current mt evaluation still faces the challenge of how to better evaluate the overlap between the reference and the model hypothesis taking into consideration adequacy and fluency, where all the evaluation units are treated the same, i.e., all the matching scores have an equal weighting. however, in real-world examinations, the questions vary in their difficulty. those questions which are easily answered by most subjects tend to have low weightings, while those which are hard to answer have high weightings. a subject who is able to solve the more difficult questions can receive a high final score and gain a better ranking. mt evaluation is also a kind of examination. for bridging the gap between human examination and mt evaluation, it is advisable to incorporate a difficulty dimension into the mt evaluation metric. in this paper, we take translation difficulty into account in mt evaluation and test the effectiveness on a representative mt metric bertscore @xcite to verify the feasibility. more specifically, the difficulty is first determined across the systems with the help of pairwise similarity, and then exploited as the weight in the final score function for distinguishing the contribution of different sub-units. experimental results on the wmt19 english↔german evaluation task show that difficulty-aware bertscore has a better correlation than do the existing metrics. moreover, it agrees very well with the human rankings when evaluating competitive systems. ## related work the existing mt evaluation metrics can be categorized into the following types according to their underlying matching sub-units: n-gram based @xcite @xcite @xcite , ## conclusion and future work this paper introduces the conception of difficulty into machine translation evaluation, and verifies our assumption with a representative metric bertscore. experimental results on the wmt19 english↔german metric tasks show that our approach achieves a remarkable correlation with human assessment, especially for evaluating competitive systems, revealing the importance of incorporating difficulty into machine translation evaluation. further analyses show that our proposed difficultyaware bertscore can strengthen the evaluation of word omission problems and generate reasonable distributions of difficulty weights. future works include: 1) optimizing the difficulty calculation; 2) applying to other mt metrics; and 3) testing on other generation tasks, e.g., speech recognition and text summarization.