O versampled ML at S em E val-2022 Task 8: When multilingual news similarity met Zero-shot approaches (2022)

Abstract: We investigate the capabilities of pre-trained models, without any fine-tuning, for a document-level multilingual news similarity task of SemEval-2022. We utilize title and news content with appropriate pre-processing techniques. Our system derives 14 different similarity features using a combination of state-of-the-art methods (MPNet) with well-known statistical methods (i.e. TF-IDF, Word Mover’s distance). We formulate multilingual news similarity task as a regression task and approximate the overall similarity between two news articles using these features. Our best-performing system achieved a correlation score of 70.1% and was ranked 20th among the 34 participating teams. In this paper, in addition to a system description, we also provide further analysis of our results and an ablation study highlighting the strengths and limitations of our features. We make our code publicly available at https://github.com/cicl-iscl/multinewssimilarity

## introduction assessing semantic similarity between two given content pieces has become one of the important natural language processing (nlp) tasks. this task can help researchers estimate the quality of their models for many other tasks such as: machine translation (mt), summarization, question answering (qa), semantic search, dialog, and conversational systems. extending semantic similarity task to a cross-lingual setup can extend the evaluation benefits to cross-lingual tasks as well. previous works @xcite focus on sentence level cross-lingual semantic similarity. the presented shared task, @xcite a novel problem that focuses on document-level semantic similarity based on news articles. the multilingual news similarity task contains monolingual as well as cross-lingual pairs of news reports. this setup enables researchers to test their multilingual models on a document-level semantic similarity task. there can be multiple extensions to this, including the clustering of news articles and tracking similarity of news coverage between different outlets or regions. in this work, we investigate the capabilities of pre-trained multilingual language models (lms) as well as word-embedding models in this task, without fine-tuning them on the task data. our solution pipeline combines pre-trained mpnet @xcite with well-known statistical methods with well-known statistical methods (i.e. tf-idf, word mover's distance) to derive the semantic similarity features between articles using their title and textual content. we use these similarity features to approximate overall similarity between article pairs. our system performance is encouraging for this task, albeit with room for improvement. in the following sections, we describe our approach and provide a detailed study of the errors made by the system. we also report the results of an ablation study highlighting strengths and limitations of our derived features. ## system overview in our approach, we formulate the multilingual news article similarity as a regression task, relying on different similarity features to approximate the overall similarity between two news reports. we choose this approach as the language pair distribution differs significantly between training set and the test set. this setup enables us to investigate the capabilities of pre-trained multilingual language models (lms) as well as word-embedding models on document-level similarity task without fine-tuning. we divide the news similarity task into a pipeline of five subtasks: article scraping, preprocessing, embedding creation, feature calculation, and inference. figure 1 illustrates the architecture of our system. ## results all the submissions for this shared task were evaluated with regard to pearson's correlation coefficient. our best performing system achieved a score of 70.1% in the evaluation phase. we were officially ranked 20 th out of 34 teams on the main task leaderboard. on the english-only subtask, we were ranked 15 th out of 34. we report scores for our top three submissions during evaluation phase in ## performance analysis in this section, we analyze the performance of feature sets used by our system and compare different subsets against each other. we also discuss some errors made by our system and possible improvements. ## conclusion in this paper, we have described our participation in the task 8 of semeval-2022, "multilingual news article similarity". we developed a system to investigate the capabilities of pre-trained language models (lms) as well as word-embedding models. our results suggest that the system performs similarly for monolingual and cross-lingual pairs, but its performance varies based on the specific language pairs. the ablation study showcases the strength of the pre-trained multilingual language models for this task. given the performance of our system, despite the noticeable variation in language-pair distributions, we speculate that our approach can be used to deliver similar results for additional languages as well. for the future, we would like to explore the system's performance with the same features but also fine-tuning our mpnet model on the training dataset. this would allow us to compare the effectiveness of pre-trained models against the finetuned model. additionally, we would like to experiment with new features such as article summary similarity and article topic similarities.