Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (2024)

Abstract: Evaluating multilingual summarization evaluation metrics, i.e., meta-evaluation, is challenging because of the difficulty of human annotation collection. Therefore, we investigate an efficient multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual meta-evaluation dataset into multilingual versions. To this end, we introduce a statistical test to verify the transformed dataset quality by checking the meta-evaluation result consistency on the original dataset and back-translated dataset. With this quality verification method, we transform an existing English summarization meta-evaluation dataset, RoSE, into 30 languages, and conduct a multilingual meta-evaluation of several representative automatic evaluation metrics. In our meta-evaluation, we find that metric performance varies in different languages and neural metrics generally outperform classical text-matching-based metrics in non-English languages. Moreover, we identify a two-stage evaluation method with superior performance, which first translates multilingual texts into English and then performs evaluation. We make the transformed datasets publicly available to facilitate future research.

## introduction the evaluation of automatic evaluation metrics, or meta-evaluation, is critical for the development and evaluation of text summarization models. however, conducting a meta-evaluation study is difficult and resource-intensive because it can require a large collection of human annotations for calculating the correlations between metric and human evaluation. this is especially true in multilingual contexts, where resources are scarce and recent advancements have predominantly centered around english. as a result, there are only a few human annotation benchmarks for summarization meta-evaluation in * equal contribution recent years @xcite @xcite @xcite , and even fewer that are multilingual @xcite @xcite . the limited availability of multilingual metaevaluation datasets makes it challenging to evaluate automatic metrics in multilingual settings. therefore, we aim to explore an efficient multilingual meta-evaluation strategy based on monolingual data. specifically, inspired @xcite , we revisit and explore the possibility of transforming a monolingual (english) meta-evaluation dataset into a multilingual dataset through neural machine translation (nmt), which can avoid the difficulty of collecting human annotations in different languages, especially the low-resource languages. to this end, our study includes two main parts: we first investigate the soundness and feasibility of our nmt-based dataset transformation approach. specifically, we identify the key question to be whether the translated dataset is of good quality and supports meaningful multilingual summarization meta-evaluation. to verify this question, we propose a dataset quality measurement strategy based on neural back-translation, or round-trip translation @xcite . specifically, we hypothesize that if a translated dataset is of good quality, then its back-translated dataset in english should yield consistent meta-evaluation results as the original english dataset for the same automatic metric. to measure this consistency, we adopt a statistical test, two one-sided test (tost) @xcite , which allows us to assess this consistency for our use case. while @xcite also applies tost for checking dataset quality, our test is fundamentally different since they aim to check whether metrics perform similarly on the translated dataset, while we focus on the back-translated dataset, which avoids the metric performance discrepancy in different languages. furthermore, to better understand the effect of translation quality in our task, we compare two nmt systems for dataset transformation, including a domain-specific nmt model, m2m-100 @xcite , and a generic large language model (llm), gpt-3.5 @xcite . our results indicate that translation quality is crucial in our context, and the data quality testing method we propose is sensitive to the performance of the translation systems employed during our dataset construction. after verifying the soundness of our proposed dataset transformation approach, we conduct a multilingual summarization meta-evaluation on the dataset transformed into 30 languages from a recently introduced english summarization metaevaluation dataset, rose @xcite . this results in a much larger dataset compared with previous work @xcite in terms of both the number of languages @xcite and data examples in the datasets. to ensure meaningful evaluation results, we use the tost results we obtained as a filtering criterion -we accept only the evaluation results that can pass the consistency test we proposed. our meta-evaluation highlights the metric performance discrepancy in different languages and reveals their limitations. furthermore, we identify a two-stage automatic evaluation approach, which first translates multilingual texts into english and then applies automatic metrics to the translated text, achieving better average performance than applying the metrics to multilingual texts directly. our contributions are three-fold: ## dataset creation to circumvent the difficulty of collecting multilingual human annotations, we apply a data creation strategy transforming existing monolingual datasets into a multilingual one using nmt systems. specifically, we choose rose @xcite , for each language, we display the iso code. we use two nmt systems to transform rose into 30 languages: (1) m2m-100, a multilingual encoder-decoder model primarily intended for translation tasks, which is chosen because it supports a wide range of languages; (2) gpt-3.5 @xcite , 2 an llm that has shown strong performance on machine translation especially when translating from and to a pivot language, e.g., english @xcite . the 30 languages we selected are listed in to evaluate the translation quality, we use roundtrip translation to translate the translated dataset back to english. we then use a few standard automatic metrics to compare the similarity between the back-translated dataset and the original english dataset. we generally observe a high variance in performance, with mostly resource-rich languages like german, italian, and french achieving a high score (e.g., rouge-2 of 70+%), while some other languages achieving lower performance (e.g., bengali, japanese and tamil with rouge-2 of 30% or less). the full results are in appendix a. the data points that fail to pass the pre-defined margin (0.1) are highlighted with bounding boxes. ## dataset analysis we design and conduct statistical analyses to verify whether our dataset obtained through machine translation can support reliable evaluation of automatic evaluation metrics. our dataset quality verification process relies on a key assumption -the meta-evaluation performed on the original dataset and the backtranslated dataset should yield consistent results if the dataset transformation is reliable. with this assumption, we leverage the two one-sided test (tost) @xcite to check this consistency property, which is frequently utilized to test whether the difference between two groups is negligible. specifically, a small difference in meta-evaluation results between the original dataset and its back-translated counterpart can indicate the maintained quality in our translated dataset. here, the meta-evaluation results are the correlations between the metric and the human evaluation results @xcite . 3 therefore, we apply tost to test the equivalence of the human-metric correlations on the original dataset (ρ orig ) and the back-translated dataset (ρ back ). to execute tost, we formulated both null and alternative hypotheses: the null hypothesis, represented by h 0 , assumes that the absolute difference between two correlations cannot be ignored when considering a predefined equivalence margin ∆ e . conversely, the alternative hypothesis (h 1 ) suggests that the difference is small enough to infer that the two correlations are nearly identical. by conducting tost, our goal is to identify the equivalence margin (∆ e ) at which we can reject the null hypothesis and conclude that two correlations are sufficiently close. for each automatic metric we are going to evaluate, we execute tost with bootstrapping (tibshirani and efron, 1993) on the datasets translated into 30 languages with kendall rank correlation as the correlation coefficient, and search for the margin ∆ e by identifying the point at which the null hypothesis can be rejected (@xmath0.05). we report the tost results on the datasets translated by m2m-100 and gpt-3.5 in figure 1 . for m2m-100, we find that the majority of the metaevaluation results cannot be regarded as equivalent under an equivalence margin (∆ e ) of 0.1. on the other hand, for gpt-3.5, the majority of the results are considered equivalent with the same margin. the test provides a systematic way to assess the translated dataset quality and highlights the importance of using a strong nmt system for the dataset transformation. therefore, we propose a data-filtering method for the actual multilingual meta-evaluation -with a pre-defined margin (∆ e ), the meta-evaluation data point (i.e., one metric evaluated on one language) that cannot pass tost will be discarded. as a case study, we set this margin to 0.1 in our study, noting that it can be adjusted based on the level of error tolerance. ## evaluating multiligual automatic summarization evaluation we now conduct a meta-evaluation of different multilingual metrics on the dataset we obtained using gpt-3.5 as the nmt system. ## conclusion in this work, we investigate an efficient multilingual summarization meta-evaluation approach by translating a monolingual dataset into a multilingual version. we develop a statistical testing procedure to verify the transformed dataset quality to ensure the validity of the meta-evaluation results. with this procedure, we compare the dataset quality translated by two nmt systems, and we found that the translation quality is critical for multilin-gual meta-evaluation. we then present a case study of transforming a monolingual meta-evaluation dataset into 30 languages with high-quality translations, and conduct a multilingual meta-evaluation of representative automatic evaluation metrics on the transformed dataset.