LSX _team5 at S em E val-2022 Task 8: Multilingual News Article Similarity Assessment based on Word- and Sentence Mover’s Distance (2022)

Abstract: This paper introduces our submission for the SemEval 2022 Task 8: Multilingual News Article Similarity. The task of the competition consisted of the development of a model, capable of determining the similarity between pairs of multilingual news articles. To address this challenge, we evaluated the Word Mover’s Distance in conjunction with word embeddings from ConceptNet Numberbatch and term frequencies of WorldLex, as well the Sentence Mover’s Distance based on sentence embeddings generated by pretrained transformer models of Sentence-BERT. To facilitate the comparison of multilingual articles with Sentence-BERT models, we deployed a Neural Machine Translation system. All our models achieve stable results in multilingual similarity estimation without learning parameters.

## introduction the assessment of similarity between documents is a central challenge in the context of information retrieval. especially the evaluation of similarities between news articles across different languages opens up opportunities for numerous downstream tasks, such as the analysis of regional differences in news coverage of topics or sentiments towards events and news. task 8 of the semeval challenge 2022 @xcite posed the problem of the assessment of similarity of news articles across a variety of languages and provided a dataset of news article pairs in seven languages, as well as a number of bilingual pairs. apart from an overall similarity score to be estimated in the challenge, a number of scores for similarity in different categories, such as narrative or entities were provided. for our submission for this task, we evaluated the performance of the word mover's distance (wmd) and sentence mover's distance (smd) in the context of similarity assessment of multilin-gual news articles. we participated in all given languages, comparing both document pairs in the same language, as well as pairs in different languages. our code is available at https://github. com/stefanjmu/semeval2022_task_8 . we considered wmd and smd due to their conceptual simplicity and capability to integrate well with resources such as pretrained word embeddings or sentence embeddings, produced by state-of-theart language models. additionally, they offer the appeal of being themselves parameter-free, and hence independent of labelled training data. the rest of the paper is organized as follows: we introduce the approaches deployed in the challenge submission, as well as the used resources, supplementing the wmd and smd. subsequently, we present the results achieved and conclude with a discussion of the results. ## system overview figure 1 shows a schematic overview of the methods we investigated for our submission. we evaluated two different methods for this task, namely the word mover's distance (wmd) @xcite and the sentence mover's distance (smd) @xcite . both approaches take two news articles and calculate a similarity score from the representation of both texts as either bag of words or bag of sentences, respectively. both approaches have been found to constitute a metric exhibiting a pronounced correlation with the humanassessed similarity scores of the text pairs @xcite . we create the required word and sentence representations using word embeddings and sentence embeddings generated from state-of-theart language models. for the wmd approach, we deployed a preprocessing pipeline, involving the tokenization of both news articles, as well as the removal of stopwords and punctuations. subsequently, the preprocessed texts were transformed into a bag of words using figure 1 : schematic overview of our experiments in order to determine the best model for submission. given two news articles, we a) use pretrained multilingual conceptnet numberbatch word embeddings for both articles and compute the word mover's distance @xcite between both non-translated texts; also b), we obtain sentence embeddings from a sentence-bert model @xcite by first translating both articles to english and then computing the sentence mover's distance @xcite on the translated documents. the vector representations provided by conceptnet numberbatch. these vector representations, together with the dot-product similarity, constituted the metric at the core of the linear optimization problem forming the word mover's distance between two texts. the wmd also allows for a different weighting of the words of texts in the evaluation of similarity. these weights were chosen according to the respective tf-idf, calculated with the help of worldlex @xcite . the smd approach required the decomposition of texts into sentences, which were subsequently encoded with a transformer language model, resulting in a bag of sentences representation of texts. by interpreting these sentence embeddings as words, a similarity score is readily computable using the wmd again. the following subsections introduce both metrics in more detail, as well as the word embeddings and models involved, which were deployed to extend the application of the metrics to similarity assessment of texts across of different languages. ## experimental setup for preprocessing routines, the python nltkpackage was utilized (version 3.6.5). the embeddings have been conducted using conceptnet numberbatch version 19.08. the tf-idf weighting used worldlex. worldlex is no longer under active development. foot_1 the transformer model was all-minilm-l12-v2 provided on sbert.net. we employed three experiments to find the best model used for submitting to the task's evaluation: 1. word mover's distance with multilingual conceptnet numberbatch embeddings 2. word mover's distance with translated news articles before embedding them using con-ceptnet numberbatch embeddings 3. sentence mover's distance using translated news articles embedded with an english sentence-bert model for the evaluation, the pearson correlation coefficient @xcite was used. the target similarity scores range from one (very similar) to four (not similar), so these scores really correspond numerically to distances. we can thus use the computed distances directly as predictions. since the calculation of the pearson correlation coefficient normalizes all predicted and actual similarity scores using the mean and standard deviation, we do not need to scale the predicted scores to the range of the labels. due to the fact that our employed methods do not need any training labels, we can directly evaluate the results on the training data provided by the task organizers. the model found to perform best was then used for the task submission. the number of articles used for each language is listed in ## results given these results, we used the sentence mover's distance with translated input texts and the pretrained language model for sentence embeddings as the model for our task submission. we achieved the 23nd place with an overall pearson correlation coefficient of 0.57. also, language pairs including german news articles are typically performing worse than average and thus decrease the overall test score in the challenge. while looking for possible reasons for such poor results, we found some articles in training and test data, which could not be extracted correctly. for instance, some german articles from the test data are identical and contain only information about the issue of opening the web pages due to privacy regulations 3 . another reason for bad performance in similarity estimation of pairs containing german article (or comparison of two german articles) could be the incompletely loaded content of some german pages, since some articles only contain the beginning of the actual text 4 , which is indicated by the spontaneous termination of the article content with the following sentence "read the full article..." 5 . in order to eliminate such externally imposed assessment impediments, the data scraping system 6 would need to be overhauled. ## discussion and conclusion for our submission, we evaluated multiple approaches, that operate on the word or sentence level and calculate a distance between two texts using a linear program optimized on pretrained word or sentence embeddings. to be able to apply englishonly models for the representation of sentences, we 3 we found two groups of articles, each with the same content: 1.group: 1586615494, 1490686353,1520406037,1524031333, 1525352422 2.group: 1572312750, 1576180076,1611845398,1612866403 ,1617051090,1619154724,1627621567, 1551767123,1562891463, etc.