G ate NLP - US hef at S em E val-2022 Task 8: Entity-Enriched S iamese Transformer for Multilingual News Article Similarity (2022)

Abstract: This paper describes the second-placed system on the leaderboard of SemEval-2022 Task 8: Multilingual News Article Similarity. We propose an entity-enriched Siamese Transformer which computes news article similarity based on different sub-dimensions, such as the shared narrative, entities, location and time of the event discussed in the news article. Our system exploits a Siamese network architecture using a Transformer encoder to learn document-level representations for the purpose of capturing the narrative together with the auxiliary entity-based features extracted from the news articles. The intuition behind using all these features together is to capture the similarity between news articles at different granularity levels and to assess the extent to which different news outlets write about “the same events”. Our experimental results and detailed ablation study demonstrate the effectiveness and the validity of our proposed method.

## introduction news article similarity measures could facilitate various important tasks such as the clustering of news @xcite , duplicate news detection @xcite @xcite , fact-checking @xcite and tracking of the spread of news @xcite . semeval-2022 task 8 @xcite assesses the similarity between news articles in terms of the real world happenings. therefore, it mainly considers the location, time, entities and narratives, instead of writing style, political spin, or tone of the article. the training set comprises of monolingual and cross-lingual news articles pairs in 10 different languages. the main contributions in this paper are: 1) we propose an entity-enriched siamese transformer whose general idea is to enable the model to explicitly learn from entity features (geolocation, or-ganization, date and quantity) that are crucial to determine the similarity between news events but difficult to extract directly from the language models during fine-tuning. 2) we explore different data augmentation approaches through semi-supervised learning in order to tackle the imbalanced data problem. 3) we compare our proposed model with strong baselines and conduct an ablation study to analyse the contribution of each component in our model. we also present an error analysis at the end of this paper. our best system which exploits language agnostic bert sentence representations @xcite ranks 2nd in the competition. ## background the goal of the task is to predict similarity scores ranging from 1 (most similar) to 4 (least similar) for a give news article pair. the training data consists of 4,964 article pairs in seven distinct languages (english, german, spanish, turkish, polish, arabic and french), with seven groups of monolingual pairs and only one group of cross-lingual pair (german and english). the 4,902 pairs in test data feature three more languages (chinese, russian and italian) and seven groups of new cross-lingual pairs. appendix a.1 (figure 3 ) shows the distribution of the train and test data. most of the news pairs exhibit low level of similarity. however, the test set contains more pairs with the same news stories. the similarity based on location, time, entities, narratives, style and tone is also annotated respectively and given for the training data. previous studies on textual similarity have investigated different approaches based on corpus, knowledge or deep neural network @xcite @xcite @xcite ), but they mainly experiment with short text pairs (e.g., the semantic textual similarity (sts) benchmark @xcite , which is a sentence-level task). mea-surement of the document similarity is arguably more challenging than that of short text since the information in the document is sparse and the model is easier to be misled by non-essential content. this paper tries to tackle the above mentioned challenges to build robust models for document similarity (section 3). ## system overview we propose an entity-enriched siamese transformer model which exploits multiple multilingual pre-trained transformers (mpt) for multilingual news document similarity. the main subdimensions of similarity as annotated by the annotators are geolocation, time, shared entities and the shared narratives between the news articles @xcite 1 . all this information is encoded in our model to capture different dimensions of news articles. figure 1 shows the architecture of our proposed model and section 3.1 discusses its details. @xcite propose a siamese and triplet network training methodology for the bert-based @xcite models to derive semantically meaningful sentence embeddings. instead of just learning the sentence representations, we use the siamese (or bi-encoder) network architecture to learn document-level representations for the purpose of capturing the narrative of the article. ## results and discussion the second part of figure 2 presents the detailed analysis of the results for the entity-enriched siamese labse trained on augmented data model (best model). we observe that model performance varies in different language settings. the model performs the worst over german-french pairs (0.619), while it achieves the highest score on french-polish articles (0.866). overall, the model performs better on mono-lingual pairs than on cross-lingual pairs. we further analyse the "serious mistakes" that our best model tends to make: giving high/low similarity scores for dissimilar/similar article pairs. we identify these instances based on the difference between the true similarity score and the predicted similarity score from the model. we examine the cases when the absolute values of the differences are larger than 2.0 (total 46 samples found), and observe that: 1) news pairs that cover different 2 ) to 0.84067 (4th best). ## conclusion we introduce an entity-enriched siamese transformer for semeval-2022 task 8: multilingual news article similarity. the error analysis shows that the entity-enrichment leads to statistically significant improvement and make models more robust for computing similarity between news articles in both monolingual and cross-lingual setting. our entity-enriched labse model achieves a pearson correlation of 0.802, ranking 2nd in the competition. the code is available at https://github. com/iknoorjobs/semeval-code . 1125 xiaojin zhu and andrew b goldberg. 2009. introduction to semi-supervised learning. synthesis lectures on artificial intelligence and machine learning, 3(1):1-130.