G lobe S umm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization (2024)

Abstract: News summarization in today’s global scene can be daunting with its flood of multilingual content and varied viewpoints from different sources. However, current studies often neglect such real-world scenarios as they tend to focus solely on either single-language or single-document tasks. To bridge this gap, we aim to unify Multi-lingual, Cross-lingual and Multi-document Summarization into a novel task, i.e., MCMS, which encapsulates the real-world requirements all-in-one. Nevertheless, the lack of a benchmark inhibits researchers from adequately studying this invaluable problem. To tackle this, we have meticulously constructed the GLOBESUMM dataset by first collecting a wealth of multilingual news reports and restructuring them into event-centric format. Additionally, we introduce the method of protocol-guided prompting for high-quality and cost-effective reference annotation. In MCMS, we also highlight the challenge of conflicts between news reports, in addition to the issues of redundancies and omissions, further enhancing the complexity of GLOBESUMM. Through extensive experimental analysis, we validate the quality of our dataset and elucidate the inherent challenges of the task. We firmly believe that GLOBESUMM, given its challenging nature, will greatly contribute to the multilingual communities and the evaluation of LLMs.

## introduction summarization is a long-standing task in natural language processing (nlp) research @xcite . in recent years, significant advancements have been made in the field thanks to the rapid development of large language models (llms) @xcite @xcite @xcite . while llms have effectively addressed many traditional text summa-rization tasks @xcite @xcite , the rapid globalization of information dissemination has created new demands for summarization techniques that can effectively summarize a large collection of event-centric multilingual news articles worldwide. events involved with armed conflicts, international relations, and political elections have always fascinated people worldwide. however, relying solely on news articles in a single language to gain an in-depth understanding of such events can be limiting. this is because news reports from different countries are often influenced by their national standpoints and cultural biases, resulting in potential distortions @xcite @xcite . to obtain a more comprehensive insight into these events, it is crucial to explore news articles from various countries and languages, allowing us to consider diverse perspectives and access more objective information. surprisingly, while advancements in llms have shown promising results in many nlp tasks, little research has been conducted for such real-world scenarios. to this end, we present the task of mcms that unifies multi-lingual, cross-lingual and multidocument summarization into a more general setting, aiming to align better with the multifaceted requirements in real-world scenarios. the goal of mcms is to succinctly capture the key information from a collection of documents written in various languages and present a cohesive summary in the target language. notably, the mcms task has three distinctive features: (1) the input consists of multiple documents, (2) the multiple documents are in different languages, and (3) the multiple documents revolve around the same event. however, the absence of a dataset that encompasses such features inhibits researchers from further study. to close this gap, we meticulously construct the globesumm dataset, which comprises the remove the same information or facts repeated in both news reports, avoid unnecessary duplication in summary. ## annotation quality assessment we next examine the superiority of our annotation method. ## conclusion to conclude, our study presents the task of mcms that unifies multi-lingual, cross-lingual and multidocument summarization to align better with the diverse needs in real-world scenarios. our benchmark, globesumm, serves this demand as the first dataset for such scenario, offering highquality summaries generated through protocolguided prompting. through experiments and analysis, conducted on outperforming llms, we unveil the shortcomings of llms in mcms and highlight the challenges of addressing redundancies, omissions and conflicts. overall, we believe globe-summ holds the potential to be used for evaluating the performance of llms in handling multi-lingual and multi-document tasks and the way we utilize protocol-guided prompting can serve as a practical case for cost-effective annotation.