Rogue Scores (2023)

Abstract: Correct, comparable, and reproducible model evaluation is essential for progress in machine learning. Over twenty years, thousands of language and vision models have been evaluated with a popular metric called ROUGE. Does this widespread benchmark metric meet these three evaluation criteria? This systematic review of over two thousand publications using ROUGE finds: (A) Critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible. (B) Differences in evaluation protocol are common, affect scores, and impact the comparability of results reported in many papers. (C) Thousands of papers use nonstandard evaluation packages with software defects that produce provably incorrect scores. Estimating the overall impact of these findings is difficult: because software citations are rare, it is nearly impossible to distinguish between correct ROUGE scores and incorrect “rogue scores.”

## introduction this work outlines a major research integrity issue that affects thousands of machine learning papers in dozens of language and vision tasks over a span of nearly twenty years. we discover that the majority of model evaluations using the benchmark rouge metric are not reproducible and that rouge scores reported in thousands of papers may be incorrect. evaluation metric integrity is critical for model development and comparison. researchers evaluate models to quantify their behaviors, successes, and failures; to compare new modeling approaches consistently against prior work; and to keep track of progress on challenging tasks. because sharing code and parameters for models is still uncommon, researchers depend on model evaluation scores reported in papers to be comparable and correct. for these reasons, systematic errors in model evaluation may have major consequences for the findings and future trajectory of entire research fields, especially for widely used evaluation metrics like rouge. 1 software and data available at: roguescores.com @xcite rouge scores are hard to reproduce. machine learning model evaluations using rouge are less reproducible than other scientific fields. model evaluations omit critical details that affect scoring, affecting the comparability of results. ## comparability we know rouge is a parameterized metric with many possible configurations, but in section 2 we learn that these configurations are frequently unreported as only 5% of papers list rouge parameters. how sensitive is rouge to these unreported configurations, and are rouge scores computed under different configurations still comparable? normally, rouge is used to measure and compare behaviors of different models. in order to probe the behavior of rouge, we do the reverse: we test 10 different rouge configurations on a single specimen model and specimen task to examine how unreported configuration affects real-world rouge scores. ## correctness thousands of papers may evaluate models using a nonstandard rouge package. we find in section 2 only 35% of papers cite a rouge package, but 76% of packages cited are nonstandard. this suggests the 755 papers in figure 3 are a small sample of 2,000+ papers using a nonstandard package. 6 surprisingly, none of these packages has been validated against rouge-1.5.5, the original rouge implementation of @xcite . this validation should have occurred years ago before these packages were ever used; but, better late than never -we will do it now. ## case studies but does it matter if evaluation is not reproducible? should we care that subtle evaluation configuration differences make results incomparable? how much do software errors actually affect evaluation? here are several concrete examples that demonstrate the real-world effects of evaluation integrity issues. ## conclusion rogue scores is the most significant and widespread research integrity issue to date in machine learning history, impacting the reproducibility, comparability, and correctness of thousands of results over a span of twenty years. we discover a large number of rouge model evaluation scores have been computed incorrectly by defective unvalidated software packages. although automated metrics like rouge cannot replace high quality human evaluation, they have an advantage of being perfectly reproducible and comparable, in theory. yet, in practice, rouge evaluation protocol is often unreported or underreported, making most rouge scores difficult to compare and impossible to reproduce. we know many rouge scores are incorrect, but missing evaluation details means we can only speculate on which ones. consequently, the validity and interpretation of thousands of results is now entirely uncertain.