Does Summary Evaluation Survive Translation to Other Languages? (2022)

Abstract: The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. The returns to such an effort would increase significantly if the dataset could be used in additional languages without repeating human annotations. To investigate how much we can trust machine translation of summarization datasets, we translate the English SummEval dataset to seven languages and compare performances across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. We also consider the effect of translation on the relative performance between measures. We find some potential for dataset reuse in languages similar to the source and along particular dimensions of summary quality. Our code and data can be found at https://github.com/PrimerAI/primer-research/ .

## introduction a large summarization dataset includes thousands of texts and human-written summaries (for example, cnn/daily mail @xcite ). in order to make it applicable for wider research, it may also contain machine-generated summaries by many models, accompanied by human and machine evaluations of the quality of the generated summaries @xcite . the human annotation alone is a complicated effort, requiring careful planning and setup @xcite @xcite . what purpose do the human annotations serve? their main utility is serving as a benchmark for automated evaluation measures. researchers design measures to closely approximate human judgment in order to increase the pace of summarization model improvement. as summarization resources grow for english-language models, it becomes increasingly important to consider whether we can repurpose these datasets for use in other languages as well. given a method that could produce flawless translations, the original human annotations quite clearly remain useful, as the relative rankings of the summaries would be invariant. in this scenario, comparing automated measures in another language with the english human scores produces valid conclusions. in practice, however, translation will introduce some distortions -both mild and extreme -that can spoil the utility of the original annotations. while a "uniform" distortion over all texts would preserve the relations among evaluation measures, this too is an unrealistic assumption as translation will correct and simplify some texts, introduce errors into others, and push components of text quality like relevance, coherence, and fluency in different directions @xcite . we are left to ask how to determine whether it is still practical to rely on the original human annotations for at least some quality measures and alternate languages? in this paper, we seek to address this question through two quantitative explorations of automated evaluation measures under translation. first, we determine how often the correlation between a given measure and the original human annotations remains equivalent under translation. second, we consider if one automated measure aligns more closely with human judgment than another in english, how often their relative positions are maintained after the translation. we conduct this investigation using the summeval dataset @xcite , the largest corpus of english-language human annotated text summaries widely available. we translate this dataset from english to seven languages and evaluate the correlations between automated summary evaluation measures and human annotations. using equivalence tests, we show that some aspects of summary quality ranking are preserved under translation for languages with similar alphabets and grammars to english. while we find some reasons for optimism about the potential for dataset reuse, our work clearly demonstrates that more research is needed to make translated datasets useful for a diverse set of languages. ## data and models we focus our analysis on the portion of summeval foot_0 that includes human annotations. it consists of 100 texts, each accompanied by 11 human-written reference summaries and 17 machine-generated summaries produced by different models. each machine-generated summary is annotated by three experts and five crowd workers using a five-point scale for four quality measures: coherence, consistency, fluency, and relevance. for simplicity, we create a composite rating by averaging the expert scores for each quality of a given text-summary pair. we translate all 100 source texts, 1100 human reference summaries, and 1700 machine-generated summaries into seven languages -french, german, italian, spanish, afrikaans, hindi, and russian -using translation models trained and uploaded to the hugging face model hub by helsinki-nlp foot_1 and accessed via the transformers library @xcite . the specific models used for translation are named 'opus-mt-l1-l2', where one of l1 or l2 is 'en' (english), and the other is one of the languages 'af', 'de', 'es', 'fr', 'hi', 'it', or 'ru'. in each language version of the dataset, we score machine-generated summaries with a few common or promising automated evaluation measures that could be applied to all eight languages. we calculate the following truly automated (not needing human written reference summaries) measures: jensen-shannon @xcite , estime @xcite foot_2 and blanc @xcite foot_3 . we also calculate the following reference-based automatic evaluation measures: bleu @xcite , bertscore-f1 foot_4 @xcite , and rouge @xcite as rouge-1,2,l foot_5 . these measures were selected to cover a wide range of strengths and weaknesses in replicating human judgment (see appendix c for more detail). we use the same original human annotations provided by the sum-meval dataset as annotations in each of the seven translated languages. we used 'bert-base-multilingual-cased' as the underlying model for blanc and estime. while other choices of underlying model could produce higher correlations with human annotations in english, this multilingual model was selected to provide a more uniform performance across languages. bertscore relies on 'bertbase-multilingual-cased' for all languages except english, for which it uses the model 'robertalarge'. estime embeddings were taken from the 10th transformer block layer instead of the final 12th layer. we followed @xcite , where it was shown that for the larger model 'bert-large-uncased-whole-word-masking' the 21st layer delivers the better performance than the 24th and final layer. we calculate correlations between automated evaluation measures in each language and the human annotations on the original english dataset. we seek to answer whether these correlations are reasonably independent of the language. in other words, can we rely on such correlations to provide consistent judgement of evaluation measures in other languages? ## comparisons between measures while our statistical tests focus on the absolute correlation between automated and human scores, we can instead consider the automated measures relative to one another. if one measure correlates better than another with human scores in the original english dataset, would it still be better in a translated (non-english) dataset? additionally, we can return the dataset back to english to get a sense of the distortion introduced by the translation process. to estimate the consistency with which one measure dominates another, we turn to bootstrap resampling of the summary evaluations. we select 10,000 bootstrap samples from the 1700 text-summaryreferences tuples. let p represent the fraction of samples in which one measure is better than another for a given measure-measure pair; we con-sider a pair "resolved" if one measure outperforms another in at least 97.5% of all the resamplings, i.e. @xmath0.975 in the original english dataset. using kendall rank correlations, the number of resolved measure-measure pairs is 64% for relevance, 61% for coherence, 56% for consistency, and 42% for fluency. with a baseline reading of how stable the measure rankings are in english, we can ask what happens with these resolved pairs when the dataset is translated. for most languages and qualities the shift of p is less than 0.1, the largest is 0.25 (consistency, hindi). many resolved measure-measure pairs become unresolved after translation, though no shift is drastic enough to reverse which measure ranks higher in a majority of samples (i.e. crossing @xmath1.5). figure 5 suggests that in most cases our conclusion about comparing two measures will not change with translation. along its x-axis, figure 5 shows how much on average the fraction p changes (increases or decreases) after translation for resolved measuremeasure pairs, where the average is over a given language and quality measure. a round-trip translation returns each summary to its source language, effectively isolating the effect of translation quality on the consistency of automated measures. here, returning to english allows us to use the evaluation measures in their original language and, for estime, blanc, and bertscore, with their original models. changes in measure performance should then reflect distortions introduced by translation while eliminating those caused by adapting measures to another language. the dashed line @xmath2. we note that the observed shifts are mostly under the diagonal -the shifts caused by translation are to some degree reversed when we return to english. the tendency of machine translation models to produce "translationese," artifacts distinguishing the output from typical human language use, is well documented (e.g. @xcite , @xcite ), so exact overlap between source and round-trip translated texts is not expected. however, automated evaluation measures rely on coarser linguistic features like word overlap and are more influenced by significant amounts of noise during the round-trip translation process. while the shifts for round-trip translations are on average smaller than for one-way, they demonstrate that translation is far from perfect and introduces enough noise to be detected by the summarization evaluation measures. notably, the points above the diagonal come from hindi, russian and afrikaans round-trip translation. this confirms our intuition that a translation to languages more distant from english is more risky for the survival of the summary evaluation. we hope further research may reveal additional ways to use the round-trip translation for the criteria of survival. ## discussion the results presented significant differences among automated summarization measures and their relationships to the four quality measures. we seek to build an intuition for these findings and make use of qualitative exploration to ground our understanding we can review the scores for the 1700 summaries in reduced dimensions using principal components analysis (pca). figure 6 shows each 1700dimensional vector projected onto the first two principal components, which collectively explain 38.5% of the variance. there are four vectors of human expert scores, corresponding to the quality measures coherence, consistency, fluency, and relevance, averaged over the three individual experts. each automated measure (for example, rouge-2) produced eight 1700-dimensional vectors, one for each language. pca can be used to disentangle the sources of divergence among evaluation measures under translation. the plot helps highlight the relative strength of translation over the summarization evaluation methods themselves. if machine translation added significant noise to the summaries, we would expect the relative position of language-specific scores in figure 6 to be inconsistent across evaluation measures. instead, we generally observe tight clusters for each evaluation measure with shared relative positions among the languages (at least when ignoring hindi and russian). this pattern reflects the "stability" of evaluation measures undergoing translation found in section 4. the pca recasts translation as a shift in geometric space; across measures, the location occupied by each language is a similar vector shift from its corresponding english point. the exercise in roundtrip translation is an indicator of reversibility for this geometric shift. the qualities and languages we note a few curious observations from figure 6 in appendix a. ## conclusion in this paper, we probed how well automated evaluations of summaries remain consistent on texts translated to other languages. we focused on the summeval dataset and considered its translation to french, german, italian, spanish, afrikaans, hindi, and russian. to answer whether english human annotations can be trusted in other languages, at least for specific qualities, we explored tests of equivalence as a gauge of consistency after translation. we found that translation can preserve correlations of evaluation metrics with the english human scores for coherence or relevance but could not conclude the same for fluency or consistency. a complete answer to our query is a challenging task, since moving to another language affects not only the dataset, but also the measures themselves. while definitely proving that the original human annotations cannot be reused is likely impossible, our results suggest that there are clear differences in performance based on the choice of target language, automated measure, and notion of quality. we call for additional research into summary evaluation metrics that can survive translation, as it offers a relatively simple path towards extending nlp capabilities for lower resource languages. future work could identify how changes in the margin of equivalence equate to deterioration of model performance. additionally, this line of research could be extended to a larger selection of languages and automated evaluation measures.