Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics (2022)

Abstract: Current practices in metric evaluation focus on one single dataset, e.g., Newstest dataset in each year’s WMT Metrics Shared Task. However, in this paper, we qualitatively and quantitatively show that the performances of metrics are sensitive to data. The ranking of metrics varies when the evaluation is conducted on different datasets. Then this paper further investigates two potential hypotheses, i.e., insignificant data points and the deviation of i.i.d assumption, which may take responsibility for the issue of data variance. In conclusion, our findings suggest that when evaluating automatic translation metrics, researchers should take data variance into account and be cautious to report the results on unreliable datasets, because it may leads to inconsistent results with most of the other datasets.

## introduction assessing the quality of machine translation (mt) systems is always crucial to promote mt research @xcite . since it is costly and time-consuming for human graders to evaluate machine translation (mt) systems, designing automatic metrics for mt has drawn booming attention during the past decades, and many metrics such as bleu @xcite and ter @xcite have been proposed consequently. generally, it is non-trivial to measure automatic metrics. conference machine translation (wmt) @xcite @xcite thereby holds the metric shared task to evaluate the performance of automatic metrics. in each year, wmt organizers collect a dataset consisting of many mt outputs annotated with human judgments, and automatic metrics are evaluated on the dataset in terms of their correlations to human judgments. over the past ten years, the official evaluation reports only independently analyzed the results of that year. to the best of our knowledge, there are no studies to put the evaluation results of ten years together and make a more systematic analysis. therefore, some key questions remain unknown: are the evaluation results consistent across different years? are the results on each dataset reliable? one may simply summarize the existing results from the official evaluation reports of the past years and answer the above questions accordingly. however, the existing results use pearson's correlation for evaluation which suffers from sensitivity to outlier data points as argued by @xcite . besides, involved metrics in the evaluation are different year by year, thus it is difficult to directly compare the results among different years. to this end, in this work, we firstly re-evaluate ten popular metrics on all available datasets in the past ten years, with the error number evaluation method @xcite . we then empirically investigate the fluctuation of metric evaluation results. surprisingly, our experiments show that the evaluation result is sensitive to the choice of datasets, which suggests that the results on some datasets may not be reliable ( §3). then we investigate two potential hypotheses about the emergence of data variance, i.e., the insignificant data points ( §4.1) and deviation of independent and identically distributed (i.i.d) assumption ( §4.2). first, we show that the data variance issue is substantially alleviated when the insignificant data points are removed. to further understand the variance that cannot be alleviated by the first hypothesis, we design a simple method to measure the distributional differences between datasets, and hypothesize that the deviation of the i.i.d assumption may contribute to the variance. for future metric evaluation, we recommend wmt community pay attention to the potential issue of data variance ## conclusion over the past ten years, the official evaluation reports of wmt metrics shared task only independently analyzed the results of that year. in this paper, we re-evaluate ten popular metrics on all available datasets in the past ten years and comparatively analyze the evaluation results among different years together. we show the problem of conducting evaluations with only one dataset. in addition, we analyze its potential reasons that the insignificant data points and deviation of i.i.d assumption may induce the issue of data variance. this fact suggests that future researches on evaluating automatic translation metrics should take data variance into account and be cautious to conclude the result on a single dataset.