Re-Evaluating Evaluation for Multilingual Summarization (2024)

Abstract: Automatic evaluation approaches (ROUGE, BERTScore, LLM-based evaluators) have been widely used to evaluate summarization tasks. Despite the complexities of script differences and tokenization, these approaches have been indiscriminately applied to summarization across multiple languages. While previous works have argued that these approaches correlate strongly with human ratings in English, it remains unclear whether the conclusion holds for other languages. To answer this question, we construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian. To measure the strength of summaries, our ratings are measured as head-to-head comparisons with resulting Elo scores across four dimensions. Our analysis reveals that standard metrics are unreliable measures of quality, and that these problems are exacerbated in Chinese and Indonesian. We advocate for more nuanced and careful considerations in designing a robust evaluation framework for multiple languages.

## introduction ensuring the functionality of large language models (llms) in a variety of languages has been important in increasing ai accessibility. many frequently used tasks and metrics for evaluating these models have been originally developed for english @xcite . this propensity towards english has led @xcite to criticize the conflation between all-natural language tasks and natural language tasks in english. in fact, @xcite note that 30-40% of acl articles do not explicitly name the languages they evaluate. the development of new multilingual llms, such as bloom @xcite and aya-23 @xcite have coincided with increased interest in models that can fluently generate text in a wide array of languages @xcite . how these models are evaluated in these languages, however, remains understudied. researchers often assume that high scores from automatic metrics indicate good summaries, because the metrics correlate strongly with human ratings. strong correlation in aggregate, however, doesn't necessarily imply that these metrics should substitute for human judgments when determining how much to trust a specific model output @xcite . anecdotally, language generation in other languages varies in quality, and current practices for evaluating llms do not easily adapt to non-english languages. automatic metrics designed to evaluate the quality of generated summaries make assumptions about scripts and tokenizations that differ drastically cross-lingually @xcite . previous works have reported varying correlation relationships between human annotation dimensions and automatic metrics @xcite @xcite . however, metrics like rouge @xcite and bertscore @xcite , while being used commonly as standard evaluation for multilingual summarization, are not carefully studied against human annotations and preferences. more recently, researchers suggest that llms such as gpt-4 can be directly used to not only generate texts but to evaluate them as well @xcite @xcite @xcite @xcite . however, this practice has not been thoroughly evaluated in languages other than english. our contributions: this paper seeks to understand whether existing metrics, namely rouge and bertscore, can be used reliably to evaluate summaries across multiple languages, and whether gpt-4 can faithfully replace human evaluators in the context of multilingual summarization. our contribution is two-fold. first, we construct a pilot dataset 1 for the evaluation of existing multilingual summarization metrics. it includes curated articles with human reference summaries in english, chinese, and indonesian. it also features pair-wise preference annotations across four evaluation dimensions, comparing human references and llm-generated summaries, for a better understanding of "what is a good summary". to rank these summaries, resulting elo scores are calculated from these pair-wise ratings. second, we revisit popular evaluation protocols (automatic metrics and gpt-4 evaluations) and analyze the correlation with human preferences using our dataset. we argue that, in english evaluations, correlations are too low in absolute terms to be used as a reliable proxy for human preferences in headto-head comparisons. these practices are further challenged when applied in multilingual settings. through our dataset contribution and correlation analysis, we caution against indiscriminately applying english evaluation frameworks to other languages. we advocate for more research on designing evaluations that explicitly account for multiple target languages. ## re-evaluation analysis we provide a breakdown analysis of the correlation within the annotation dimensions and between the dimensions and automatic evaluation methods. we report r 2 values, the square of the pearson correlation for the analysis. foot_2 ## discussion and conclusion in this work, we discuss the weaknesses of automatic summarization evaluation methods. current assessment practices prioritize english, and use the likert scales to collect human ratings, which potentially lead to skewed and uninformative annotation distributions. instead, we propose to use pair-wise comparisons as a more fine-grained ranking approach, using elo as our metric, and intro-duce a pilot dataset for english, chinese, and indonesian. our findings indicate that human subjective preferences are not highly correlated with fluency and accuracy across languages, similar to criticisms identified in english ratings in @xcite . we measure how well automatic evaluation predicts these human ratings with both r 2 values and mean absolute error (mae), as opposed to other correlation metrics used in prior evaluation work. r 2 allows us to more stringently characterize how much of the variability in elos scores is captured by an automatic metric. additionally, mae grounds the differences in human preferences and automatic metrics, for the average example summary, within the scale of the elo scores. as in previous work @xcite @xcite , we also observe that gpt-4 ratings do not align with human preferences, showing a higher preference for llm summaries. therefore, we argue that future metrics should be designed to better predict human preferences and generalize across multilingual settings @xcite .