EMBEDDIA at S em E val-2022 Task 8: Investigating Sentence, Image, and Knowledge Graph Representations for Multilingual News Article Similarity (2022)

Abstract: In this paper, we present the participation of the EMBEDDIA team in the SemEval-2022 Task 8 (Multilingual News Article Similarity). We cover several techniques and propose different methods for finding the multilingual news article similarity by exploring the dataset in its entirety. We take advantage of the textual content of the articles, the provided metadata (e.g., titles, keywords, topics), the translated articles, the images (those that were available), and knowledge graph-based representations for entities and relations present in the articles. We, then, compute the semantic similarity between the different features and predict through regression the similarity scores. Our findings show that, while our proposed methods obtained promising results, exploiting the semantic textual similarity with sentence representations is unbeatable. Finally, in the official SemEval-2022 Task 8, we ranked fifth in the overall team ranking cross-lingual results, and second in the English-only results.

## introduction detecting news stories related to a single theme and combining them into news clusters has been an increasing interest in the creation of news aggregators that consolidate thousands of articles from different publishers and websites @xcite . tracking similarity of news coverage between different outlets or regions has also been urgent and challenging. for example, whether previously with ebola or recently with the covid-19 pandemic, monitoring and containment of infectious disease outbreaks has remained a key component of public health strategy to contain the diseases. the ability * * equal contribution from all the authors. to track disease outbreaks in an accurate manner is critical in the deployment of efficient intervention measures. as such reports may not only be in english, there is also a need for effective multilingual systems. hence, recent research has been focused on the area of identifying similarities between documents, phrases, stories, etc. semantic textual similarity (sts) deals with determining how similar two groups of sentences are by measuring their semantic similarity. over the years, several solutions were proposed to assess sts. the most general approach is pre-training on massive datasets before fine-tuning on subsequent downstream tasks @xcite @xcite @xcite . other works considered finding the similarity by classifying texts using bert-based models @xcite with a pair of sentences packed together as input @xcite @xcite . the semeval-2022 task 8 (multilingual news article similarity) @xcite aimed at developing systems that identify multilingual news articles that provide similar information by rating them on a real-valued [1 -4] scale, from most to least similar. in this paper, we cover several techniques and propose different methods for finding the multilingual news article similarity by exploring different aspects of the dataset. we consider that the textual content, the provided metadata (e.g. title, keywords, topics), representative images corresponding to the news articles, and knowledge graph-based representations for entities and relations present in the articles, would draw on a multiplicity of modes, all of which contribute to the meaning and the main story of the news articles. moreover, we also translate the articles to a high-resource language (english) in order to assess the ability of our models in an english-only context. therefore, we investigate the multimodality of the data by experimenting with sentence, image, and knowledge graph embeddings in two scenarios: (1) by directly computing the semantic similarity between the different features and (2) by learning through regression and predicting the similarity scores. ## data the training data has 4,964 article pairs from seven languages (english, german, spanish, arabic, polish, turkish, and french) and gold standard similarity scores for six dimensions (geography, entities, time, narrative, style, tone), plus the overall score. the final evaluation data has 4,902 pairs and three "surprise" languages that were not present in the training data (chinese, italian, and russian) foot_0 . moreover, the metadata includes the article titles, several specific topics and keywords, and links to representative images. the statistics of the training and final evaluation data are in ## experiments next, we detail all our approaches and perform a detailed error analysis foot_1 . the evaluation is performed in terms of pearson correlation. our results are presented in ## error analysis semantic textual similarity we can substantially improve the english-only model (1d) for sts by fine-tuning not just with monolingual english pairs from the training data but by using all the machine-translated pairs. however, we observe some cases where our best performing fine-tuned model is misled by similar turns of phrase even if the article pair covers different events. we show extracts from an article pair in article1 article2 1492472369 (en): at least one person has been confirmed dead, following saturday's fire that gutted the mgbuka obosi spare parts market in idemili north local government area of anambra . . . mr edwin okadigbo, the public relations officer of the nigeria security and civil defence corps (nscdc), anambra command . . . 1530831511 (en): at least, one person has been confirmed dead . . . in a road mishap that involved a commercial bus and a motorcycle in mbosi junction, ihiala local government area of anambra state on tuesday . . . spokesperson of the nigeria security and civil defence corps, nscdc in anambra state, edwin okadigbo said preliminary . . . ## conclusions in this paper, we covered several techniques for assessing the similarity between multilingual and monolingual news articles in the context of semeval-2022 task 8 multilingual news article similarity. we notice that, even if using images and knowledge graph representations give promising results, approaching sts with sentence embeddings is still unbeatable. however, images, being a language-agnostic medium, could be helpful if they depict people or text boxes. future work could include an adaptable inclusion of images (for handling missing images) and the usage of multilingual knowledge graph representations.