D ata S cience-Polimi at S em E val-2022 Task 8: Stacking Language Models to Predict News Article Similarity (2022)

Abstract: In this paper, we describe the approach we designed to solve SemEval-2022 Task 8: Multilingual News Article Similarity. We collect and use exclusively textual features (title, description and body) of articles. Our best model is a stacking of 14 Transformer-based Language models fine-tuned on single or multiple fields, using data in the original language or translated to English. It placed fourth on the original leaderboard, sixth on the complete official one and fourth on the English-subset official one. We observe the data collection as our principal source of error due to a relevant fraction of missing or wrong fields.

## introduction semeval-2022 task 8 @xcite (multilingual news article similarity) is a document-level similarity task on news articles data. the goal is to predict whether two multilingual news articles cover the same real-world happening regardless of their writing style, political spin and tone. the task included resources written in 10 different languages: english, german, spanish, turkish, polish, arabic, french, chinese, italian and russian (the training dataset included news written only in the first seven languages). this task is interesting as we can apply the obtained approaches to cluster news articles and track the similarity of news coverage between different outlets or regions as done in the agenda setting project 1 . our best model is a simple but effective stacking of a set of language models trained on different combinations of textual features. we fine-tune 14 language models, half of them with original multilingual texts and half with texts translated to english. we select three textual fields from the features extracted by our scraper (title, body and description of the news article), and we fine-tune a model for every combination. our model mainly struggles with missing or wrong data since the training and evaluation datasets were released as links to scrape due to privacy policies. we noticed that we could not collect parts of the datasets, and some of the collected data were clearly wrong (e.g. "get in touch with us. all rights reserved" as the body of an article). the code will be available on github foot_0 . ## results and ablation however, we obtained our best submitted prediction using our best performing model on the validation set: a stacking of all single-field and multiple-fields lms, with the same final estimator as before. this model allowed us to score fourth in the original leaderboard that includes the best performing model for each team, and sixth on the official ranking 9 . the second part of the table reports the results of the six double-fields lms and the two triplefields lms. combinations and interactions of fields give performances worse than our best performing single-field lms (mpnet on body). we believe that this degradation of performance is due to two main factors. first, the noisiness of the title and description, as a higher percentage was missing or wrong. second, due to memory issues, we had to perform multiple-field lms training reducing the maximum length of the body to 256 tokens instead of 512, as used when training single-field lms. the third part of the table shows performances of our best four stacking models. we evaluated many final estimators and combinations of first estimators and we submitted the scores from the models that performed better on our validation set. we computed the validation performance of stacking models using 5-fold cross-validation on the original validation dataset. finally we report the performance of the best team that participated to the competition. up to now we do not know details about their approach. ## conclusion to quantify the similarity between news articles, we propose an approach trained exclusively on the extracted textual data. we initialize our architecture with sota semantic-similarity language models, which we fine-tuned on titles, descriptions and texts of the articles. we also design a simple variant to process many textual fields at once. finally, we perform stacking with a simple mlp, as it was proved to improve the overall performance of models trained on different features. results show how the approach successfully estimated similarities since the main sources of error involved missing or wrong data.