Reproducibility Issues for BERT -based Evaluation Metrics (2022)

Abstract: Reproducibility is of utmost concern in machine learning and natural language processing (NLP). In the field of natural language generation (especially machine translation), the seminal paper of Post (2018) has pointed out problems of reproducibility of the dominant metric, BLEU, at the time of publication. Nowadays, BERT-based evaluation metrics considerably outperform BLEU. In this paper, we ask whether results and claims from four recent BERT-based metrics can be reproduced. We find that reproduction of claims and results often fails because of (i) heavy undocumented preprocessing involved in the metrics, (ii) missing code and (iii) reporting weaker results for the baseline metrics. (iv) In one case, the problem stems from correlating not to human scores but to a wrong column in the csv file, inflating scores by 5 points. Motivated by the impact of preprocessing, we then conduct a second study where we examine its effects more closely (for one of the metrics). We find that preprocessing can have large effects, especially for highly inflectional languages. In this case, the effect of preprocessing may be larger than the effect of the aggregation mechanism (e.g., greedy alignment vs. Word Mover Distance).

## introduction reproducibility is a core aspect in machine learning (ml) and natural language processing (nlp). it requires that claims and results of previous work can independently be reproduced and is a prerequisite to trustworthiness. the last few years have seen vivid interest in the topic and many issues of non-reproducibility have been pointed out, leading to claims of a "reproducibility crisis" in science @xcite . in the field of evaluation metrics for natural language generation (nlg), the seminal work of @xcite has demonstrated how different preprocessing schemes can lead to substantially different results when using the dominant metric at the time, bleu @xcite . thus, when researchers employ such different preprocessing steps (a seemingly innocuous decision), this can directly lead to reproducibility failures of (conclusions regarding) metric performances. even though bleu and similar lexical-overlap metrics still appear to dominate the landscape of nlg (particular mt) evaluation @xcite , it is obvious that metrics which measure surface level overlap are unsuitable for evaluation, especially for modern text generation systems with better paraphrasing capabilities @xcite . as a remedy, multiple higher-quality metrics based on bert and its extensions have been proposed in the last few years @xcite . in this work, we investigate whether these more recent metrics have better reproducibility properties, thus filling a gap for the newer paradigm of metrics. we have good reason to suspect that reproducibility will be better: (i) as a response to the identified problems, recent years have seen many efforts in the nlp and ml communities to improve reproducibility, e.g., by requiring authors to fill out specific check lists. foot_0 (ii) designers of novel evaluation metrics should be particularly aware of reproducibility issues, as reproducibility is a core concept of proper evaluation of nlp models @xcite . our results are disillusioning: out of four metrics we tested, three exhibit (severe) reproducibility issues. the problems relate to (i) heavy use of (undocumented) preprocessing, (ii) missing code, (iii) reporting lower results for competitors, and (iv) correlating with the wrong columns in the evaluation csv file. motivated by the findings on the role of preprocessing and following @xcite , we then study its impact more closely in the second part of the paper (for those metrics making use of it), finding that it can indeed lead to substantial performance differences also for bert-based met-rics. the code for this work is available at https: //github.com/cyr19/reproducibility. ## related work relevant prior work to this work includes bertbased evaluation metrics (section 2.1) and reproducibility in nlp (section 2.2). ## datasets & metrics in our reproduction experiments (section 4), following zhang et al. (2019), zhao et al. (2019) and colombo et al. (2021) , we use wmt15-18 @xcite @xcite for mt evaluation. besides, we follow @xcite to use tac2008 foot_1 and tac2009 foot_2 for text summarization evaluation, mscoco @xcite for image captioning (ic) evaluation, and bagel @xcite and sfho-tel @xcite for data-to-text generation (d2t) evaluation. for the reference-free metric sentsim, we will mainly report results on the mlqe-pe dataset @xcite . in further experiments (section 5), we consider wmt19 @xcite for mt as well. the datasets for each nlg task are described in detail in the appendix (section a.1). for our reproduction attempts, we consider moverscore, bertscore, baryscore, and sentsim. metrics moverscore measures semantic similarity between reference and hypothesis by aligning semantically similar words and computing the distance between these words using the word mover distance @xcite . bertscore calculates the cosine similarity (of bert representations) for each token in the reference with each token in the hypothesis, and uses greedy alignment to obtain the similarity scores between sentences. it has three variants: recall, precision, and f1. baryscore computes the wasserstein distance (i.e., earth mover distance @xcite ) between the barycentric distribution @xcite of the contexual representations of reference and hypothesis to measure the dissimilarity between them. sentsim has both reference-free and -based versions; we experiment with its referencefree version in this work, which combines sentence-(based on @xcite ) and wordlevel models (extending a.o. bertscore to the multilingual case) to score a pair of source text and hypothesis. ## reproduction attempts our main focus will be to reproduce the results on machine translation (mt) reported in @xcite al. (2019), zhao et al. (2019), colombo et al. (2021) and @xcite . ## sensitivity analysis in the previous section, we have seen that preprocessing may play a vital role for obtaining stateof-the-art results (at least for some of the metrics). similar to the case of bleu @xcite , we now examine this aspect in more detail. according to the papers and evaluation scripts, moverscore uses the following main preprocessing steps (besides those handled by bert): (i) subwords removal: discard bert representations of all subwords except the first. (ii) punctuation removal: discard bert representations of punctuations. (iii) stopwords removal: discard bert representations of stopwords (only for summarization). 9 the preprocessing steps for bertscore and baryscore are only related to lowercasing and tokenization, both of which are handled by bert. we observe that (i) moverscore uses much more preprocessing than bertscore and baryscore on wmt datasets; (ii) authors may take different preprocessing steps for different tasks, e.g., @xcite remove stopwords for summarization but not for mt. besides preprocessing in a narrower sense, all three considered evaluation metrics use parameters. this makes them more flexible, but also complicates reproduction: the difference in one parameter setting can lead to reproduction failure. we study the impact of the parameters related to idfweighting. idf-weighting measures how critical a word is to a corpus; thus, it is corpus-dependent. the choice of corpus may lead to deviations of metric scores. moverscore is the main experiment object in the remainder. compared to the other metrics, its authors took more preprocessing steps to achieve the results in their paper, suggesting that it is more likely to obtain uncomparable scores across different users when using moverscore. we will also investigate the sensitivity of bertscore to the factors discussed above; we omit baryscore and sentsim from further consideration. impor-tantly, we move beyond english-only evaluation, as reported in the original moverscore paper. this will estimate how much uncertainty there is from preprocessing when a user applies moverscore to a non-english language pair, which requires new idf corpora, new stopword lists and may have higher morphological complexity (which is related to choice of subwords). we use two statistics to quantify the sensitivity of the evaluation metrics. when there are only two compared values a, b, we compute relative difference (rd) to reflect the relative performance variation regarding a certain parameter. when there are more than two compared values, we compute coefficient of variation (cv) to reflect the extent of variability of the metric performance: we only consider mt and summarization evaluation in this part. in each experiment, we only adjust the settings of the tested factors and keep the others default (given in section a.5). in addition to english ("to-english"), we consider mt evaluation for other 6 languages ("from-english"), for which we use multilingual bert: chinese (zh), turkish (tr), finnish (fi), czech (cs), german (de), and russian (ru). note that in these cases, we compare a chinese reference to a chinese hypothesis and analogously for the other languages. ## conclusion we investigated reproducibility for bert-based evaluation metrics, finding several problematic aspects, including using heavy undocumented preprocessing, reporting lower scores for competitors, selective evaluation on datasets, and copying correlation scores from wrong indices. our findings cast some doubts on previously reported results and findings, i.e., whether more the complex alignment schemes are really more effective than the greedy alignment of bertscore. in terms of preprocessing, we found that it can have a large effect depending (a.o.) on the languages and tasks involved. for a fairer comparison between metrics, we recommend to ( 1 ) additionally report the results on the datasets that the competitors used, (2) check whether the used versions of the competitor metrics can obtain comparable results as in the original papers, and (3) minimize the role of preprocessing (ideally employing uniform preprocessing across metrics). on the positive side, as authors are nowadays much more willing to publish their resources, it is considerably easier to spot such problems, which may also be one reason why critique papers such as ours have become more popular in the last few years @xcite . in a wider context, our paper contributes to addressing the "cracked foundations" of evaluation for text generation @xcite and to better understanding their limitations @xcite . in the future, we would like to reproduce more recent bert-based metrics -e.g., with other aggregation mechanisms @xcite , normalization schemes @xcite , different design choices @xcite , or metrics that use supervision @xcite @xcite -to obtain a broader assessment of reproducibility issues in this context. we would also like to quantify, at a larger scale, the bias in research induced from overestimating one's own model vis-à-vis competitor models. limitations of our work include (1) a limited number of explored evaluation metrics, (2) a restricted focus on mt only and (3) reliance on authorprovided reproduction resources.