R epro H um #0067-01: A Reproduction of the Evaluation of Cross-Lingual Summarization (2025)

Abstract: Human evaluation is crucial as it offers a nuanced understanding that automated metrics often miss. By reproducing human evaluation, we can gain a better understanding of the original results. This paper is part of the ReproHum project, where our goal is to reproduce human evaluations from previous studies. We report the reproduction results of the human evaluation of cross-lingual summarization conducted by (CITATION). By comparing the original and reproduction studies, we find that our overall evaluation findings are largely consistent with those of the previous study. However, there are notable differences in evaluation scores between the two studies for certain model outputs. These discrepancies highlight the importance of carefully selecting evaluation methodologies and human annotators.

## introduction in recent years, natural language processing (nlp) has witnessed remarkable progress, driven by advances in nlp models and data sources. this progress has led to significant improvements across a wide range of nlp tasks, including machine translation @xcite ), text summarization @xcite , reasoning @xcite , and question answering @xcite . evaluation plays a crucial role in assessing nlp models before they are deployed in real-world applications @xcite . nlp model evaluation is typically conducted using automated metrics such as bleu @xcite and rouge @xcite . in addition to these metrics, human evaluation also plays an important role by providing insights into model performance based on human preferences and real-world applicability. reproduction studies are crucial for ensuring the reliability and quality of research experiments, * corresponding author. as part of the reprohum project b batch experiment @xcite , we focus on reproducing the human evaluation conducted in the paper "cross-lingual abstractive summarization with limited parallel resources" by @xcite . the original study aims to improve cross-lingual summarization in low-resource settings. specifically, for the human evaluation, they assessed 60 chinese paragraphs with four different english summarization results each. in this paper, we first detail the experiments conducted in the original research, with a specific focus on human evaluation in section 2. we then introduce our reproduction setting in section 3. finally, we report the quantified reproducibility assessment and compare the results of our reproduction study with those of the original study in section 4. ## original study the study we are focusing on reproducing is "cross-lingual abstractive summarization with limited parallel resources" by @xcite . in the original study, the authors proposed multi-task cross-lingual abstractive summarization (mclas), a framework designed to enhance crosslingual summarization in low-resource settings. the model employs a pre-training and fine-tuning strategy. initially, it is pre-trained on a large-scale monolingual document-summary dataset to equip the decoder with general summarization capabilities. subsequently, it is fine-tuned on a small num-ber of parallel cross-lingual summary samples to transfer the learned summarization capabilities to low-resource languages. ## reproduction settings in this study, we focus on reproducing the human evaluation from the original study. we express our gratitude to the original authors for sharing the experiment data, from the evaluation forms and the anonymized annotation results. with this data, we can compare our reproduction results with the original study. we filled human evaluation datasheet (heds), a document containing the comprehensive details for the human evaluation reproduction experiment. the heds document is available in a github central repository. foot_0 ## quantified reproducibility assessment the evaluation in this study follows the standardized procedure established by the reprohum project, which categorizes reproducibility into four types of results @xcite . in type i, we report the single numerical scores and coefficient of variation (cv) values. for type ii, we calculate both pearson and spearman correlation coefficients. in type iii, we present an agreement score that quantifies the level of alignment between the original and reproduced results. finally, for type iv, we provide the comparison of conclusions and key findings from both the original and reproduction experiments. type i first, we report the score human evaluation result using best-worst scaling method. we report the score of original experiment and our reproduction experiment. next, we calculated the coefficient of variation (cv) values for each model across different scenarios to assess the precision of the results. following belz (2022), we adjusted the cv for small sample sizes, referring to this adjusted value as cv*. since the measurements included negative values, we shifted the measurement scale by adding 1 to ensure all values were claim claim 1: as the data size increases, all the models achieve better results. claim 2: mclas outperformed ncls and ncls+ms in all the metrics claim 3: mclas is especially strong in conciseness. our findings are similar to the previous study, showing that ncls is the best model in the minimum scenario, while mclas is the best model in the medium and maximum scenarios. however, in some results, only the maximum scenario has a low cv* score, which lower cv* score represents better result. this indicate that only the reproduction results of the maximum scenario are close to the original study. type ii next, we report the correlation between original and reproduction study using pearson and spearman correlations. the result is presented in type iii next, we report the krippendorff's α value from the original and reproduction annotation results. we report almost all of the models have low values of krippendorff's α. these shows the less agreement between original and reproduction study for each annotations. type iv finally, we report whether the findings from the original experiment were verified in our reproduction study. the original study claimed three key findings, listed in ## discussion from the results, we conclude that the reproduction findings align with the original study. in the minimum scenario, the best model is ncls, while for the medium and maximum scenarios, the best model is mclas. however, the correlation scores indicate only slight agreement. we hypothesize that this may be due to annotator quality, as we recruited master's students studying nlp. if we had chosen experts in both chinese and english language, the annotation quality might have been significantly better. when reviewing the annotations, we noticed that some annotators occasionally scored the models inconsistently in a small occurence. for instance, in a single paragraph, two or three models output might be labeled as worst (-1) or best (1). this inconsistency arose because the annotation platform did not restrict such settings. to address this, we contacted the annotators with these issues and asked them to reannotate the data manually, providing them with the correct annotations as a reference. surprisingly, we also found this errors in original study, where there is a participant score two models as best (1). additionally, upon reviewing the incorrect an-notations, we suspect that the best-worst scaling method may not be the most appropriate option for rating these outputs. as illustrated in figure 2 , the outputs from models 3 and 4 are both uninformative and provide incorrect information within the paragraph. this may lead to confusion for the annotators when selecting only one result to be marked as the worst. we suggest that it might be more effective to use a different approach to evaluate the models, such as rating each result on a scale from worst to best (1-5). from these findings, we recognize the critical importance of annotator quality in achieving consistent evaluation, especially when dealing with multiple languages. we also understand that the choice of evaluation methodology significantly impacts the quality of the results. ## conclusion in this study, we report our reproduction experiment from paper "cross-lingual abstractive summarization with limited parallel resource". we reproduce the human evaluation with the similar setup as the original paper reported, but we only evaluate one metric instead of three by following the instructions from the organizer. by comparing the results between original and reproduction study, we found that the scores differs in several models. this highlights the importance of the choice of evaluation methodology and evaluators.