Multifaceted Challenge Set for Evaluating Machine Translation Performance (2023)

Abstract: Machine Translation Evaluation is critical to Machine Translation research, as the evaluation results reflect the effectiveness of training strategies. As a result, a fair and efficient evaluation method is necessary. Many researchers have raised questions about currently available evaluation metrics from various perspectives, and propose suggestions accordingly. However, to our knowledge, few researchers has analyzed the difficulty level of source sentence and its influence on evaluation results. This paper presents HW-TSC’s submission to the WMT23 MT Test Suites shared task. We propose a systematic approach for construing challenge sets from four aspects: word difficulty, length difficulty, grammar difficulty and model learning difficulty. We open-source two Multifaceted Challenge Sets for Zh→En and En→Zh. We also present results of participants in this year’s General MT shared task on our test sets.

## introduction machine translation (mt) evaluation is an indispensable part of mt research, helping researchers verify the effectiveness of proposed training strategies and offering suggestions for future researches. however, automatic machine evaluation has raised a lot of concerns during decades of practices. one research direction is to explore the weakness of available evaluation metrics @xcite @xcite . another direction is to analyze the soundness of test sets. for example, freitag et al. (2020) discuss the impact of reference translationese on the evaluation results. however, to our knowledge, few researches @xcite has been done to discuss the influence of source sentences on the evaluation results. with the advancement of machine translations in recent years, we think that randomly sampled test sets may not be able to reflect the true gaps among models, as you can't test freshman's capability with grade-1 quiz. so we propose a strategy to collect test sentences with high-level of difficulty. the strategy considers a sentence's difficulty level from four dimensions, including word difficulty, length difficulty, grammar difficulty and model learning difficulty. this paper presents our constructed multifaceted challenge sets 1 for @xmath0. each of the test set contains 2,000 sentences. the source sentences are from the open-sourced english wikipedia corpus 2 while the translations are provided by our in-house translators. we report the results of participants in this year's general mt shared task on our test sets and hope to gain some insight by comparing our results with the official evaluation results. ## conclusion and limitations this paper presents hw-tsc's submission to the wmt23 mt test suites shared task. we propose increasing the test set difficulty level to better measure model performances. we propose a strategy to collect test sets with high difficulty level: word difficulty, length difficulty, grammar difficulty and model learning difficulty. we construct two multifaceted challenge sets for @xmath1. however, due to time constraints, we do not perform human evaluations on the test results, which we believe will offer more insights on the performance of our challenge sets. for future researches, we will conduct direct assessment (da) and error annotations to explore the performance of each participants on the challenge sets and compare the result with the official test sets. in addition, we will construct relatively simple test sets in the same domain, and compare the results with these challenge sets, hoping to gain more insights on the role of source sentence difficulty level.