S em E val-2022 Task 8: Multilingual news article similarity (2022)

Abstract: Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.

## introduction consider the following question: given a pair of "hard" news articles, 1 are they covering the same news story? answering this question likely requires knowing specific aspects of the events covered: what happened, where and when it happened, who was involved, and why and how it happened @xcite @xcite ). effectively modeling the similarity of news stories holds substantial practical benefits in structuring the content of the hundreds of thousands of news articles generated every day. 2 given the volume of articles, an effective measure of news story similarity enables clustering and identification of event coverage in news media @xcite . commercial news aggregation services, as provided by, e.g., google news or apple news, perform a similar clustering approach, yet are primarily monolingual and have not been made openly available or extensively researched beyond proprietary solutions. in addition, quantifying news article similarity allows the comparison of news outlets in terms of their coverage, understanding which stories consume much of the media agenda, as well as tracking the diffusion of news stories through a media ecosystem and over time. being able to measure these aspects is important for a host of research questions in media and communication studies including, for example, agenda setting @xcite . another highly desirable property of such methods is to be applicable in multilingual settings, to detect news stories covered across languages in an increasingly globalized news ecosystem @xcite . assessing the similarity of two news articles introduces new challenges not found in traditional semantic textual similarity. most importantly, methods for semantic textual similarity typically measure the extent to which two arbitrary documents are "the same," without concretely specifying the meaning of this similarity, or only do so in broad strokes (cf. @xcite @xcite . one byproduct of this vague application domain and under-specification is that agreeing on gold labels is notoriously difficult, at least at the full-length document level @xcite , as even human labelers are dependent on specific instructions and/or knowledge of the absolute space of documents to label, to understand the relative concept of "similar" @xcite . news article similarity is thus more related to attempts to compare narratives @xcite @xcite , which require understanding the structure and content to assess similarity. here, we introduce semeval 2022 task 8 for the task of quantifying news similarity, a hard discourse-level task. stories often include a variety of descriptions, people, and entities that may appear in another, dissimilar story. further, the temporal nature of news means that as real-life events evolve, stories describing the same event may include new details or entities-possibly becoming a new news story altogether. for this task, we create a high-quality dataset by annotating pairs of news article for similarity in 10 different languages on several dimensions, e.g, geographic, temporal, and narrative similarity. participants in this task were given a large collection of news articles, with 4,918 pairs receiving ground-truth similarity labels, and were asked to estimate the overall similarity of 4,902 news article pairs given to participants without labels for any dimension. the task is also challenging due to its large language diversity: the training data consists of 8 language combinations, while the evaluation dataset has 18 language combinations including three languages not appearing at all in the training data. ## results the task received over 500 public submissions from over 30 participants. next, we provide an overview of the baselines and the approaches that have been adopted by the 19 teams who participated in the competition's leaderboard and submitted a description of their systems. ## discussion multilingual news article similarity is a challenging problem despite sharing some characteristics with semantic text similarity. participants in this sem-eval task tried a number of innovative approaches to the problem. systems that used multiple parts of the article (headline, body, publication date) and systems that fine-tuned or otherwise trained embeddings generally performed better than those that did not. the best-performing systems generally combined multilingual embeddings and translation. nonetheless, there was no clear consensus as to the best architectures, embedding models, or preprocessing to perform on the data. there were clear variations across languages, and more work is needed to create multilingual systems that work across diverse language combinations. errors were particularly common when the news articles shared some similarity in terms of their geographic focus, temporal focus, named entities, and narratives but were nonetheless dissimilar overall. while the best-submitted model achieved a very high correlation of 0.82 with gold standard labels, the best human annotator reached 0.88 correlation, which suggests ample space for further progress. 9 our dataset is drawn from the first half of 2020 and covers several geopolitical events (e.g., blacklivesmatter) as well as the first wave of the covid-19 pandemic. the nearly 10,000 annotated pairs of news articles across 18 combinations of 10 different languages will enable exciting developments in natural language processing methods as well as social science studies of how the global media reported on this unique period. table a1 : rankings for each team in the official mulingual setting and in the optional english-only setting, in which one additional team participated. the final team rankings shown here were computed through the bootstrapping process described in §3.3. we additionally report the maximum and mean scores (pearson r) for each team, which largely correspond to the same ranking as our bootstrapping process.