W ue D evils at S em E val-2022 Task 8: Multilingual News Article Similarity via Pair-Wise Sentence Similarity Matrices (2022)

Abstract: We present a system that creates pair-wise cosine and arccosine sentence similarity matrices using multilingual sentence embeddings obtained from pre-trained SBERT and Universal Sentence Encoder (USE) models respectively. For each news article sentence, it searches the most similar sentence from the other article and computes an average score. Further, a convolutional neural network calculates a total similarity score for the article pairs on these matrices. Finally, a random forest regressor merges the previous results to a final score that can optionally be extended with a publishing date score.

## introduction the goal of the multilingual news article similarity task @xcite is to check pairs of multilingual news articles against each other in terms of similarity of their information content. the challenge focuses on what is talked about (time, geolocation, shared entities), not how the information is expressed (writing style, emotional tone, etc.), which is essential for applications such as analyzing the news coverage between different regions. the participants' objective is the creation of a model that rates the similarity of article pairs on a 4-point scale from most (1) to least (4) similar and achieves the highest possible pearson correlation score compared to the gold standard. the languages covered by this competition are english (en), german (de), spanish (es), turkish (tr), polish (pl), french (fr) and arabic (ar) in the training, additionally italian (it), russian (ru) and chinese (zh) in the evaluation. our system uses an ensemble approach to score pair-wise sentence similarity matrices of the article pairs, which are created with sbert and universal sentence encoder sentence embeddings. scores are obtained through simple matrix operations and our convolutional neural network simcnn based on the textcnn by @xcite . finally, a random forest regressor @xcite consolidates the individual scores into a final result, which we extend with a publishing date score. the key challenge of this task is the usage of multilingual text pairs, which requires the application of less precise multilingual language models. also the splitting of sentences and named entity recognition becomes hard to accomplish, since models for these tasks are still monolingual in most cases and we found the few multilingual ones to be unreliable. the article scraping tool provided by the authors also did not reveal meaningful features to work with besides the article title and text, since the scraped keywords, tags and publishing dates were not always available and in a utilizable format. in the competition, we took 9th place with a pearson correlation coefficient of 0.759 on the evaluation set, which differs from the observed performance on our own validation sets. this is due to a shift from latin languages to more complex languages like chinese in the evaluation data, where the performance of the multilingual models and sentence tokenizing algorithm decreases. our code is publicly available foot_0 . ## background the starting point of the task is the training dataa csv file -which contains a list of article pairs foot_1 . each article pair consists of the languages, ids and urls for both articles and the gold standard similarity scores for multiple aspects of the articles (geography, entities, time, narrative, style, tone, overall), of which only the overall score is relevant for our evaluation. to get the content and metadata of the articles, the task authors provide a scraping tool foot_2 utilizing the newspaper3k python package that downloads the html page and creates a json file that contains the title, text, keywords, labels, date, and more properties of each news article. for the training of the prediction system on the json files, the results can be compared against the overall scores in the initial csv file. the final evaluation data csv, for which the results are submitted, does not provide scores to compare against. ## related work for the scoring of our similarity matrices in subsection 4.4, we use an idea from @xcite , who introduce self-supervised document similarity ranking (sdr), an unsupervised approach built upon the roberta language model to rank the semantic similarity of a collection of documents to a source (query) document. sdr captures the intuitive fact that for each sentence or paragraph in one document, there should be at least one similar one anywhere (obtained by a max operation on the rows and columns of the similarity matrix) in the other document if both deal with the same topic. since sdr is monolingual and is only trained on english texts, we combine its scoring approach with multilingual embeddings obtained from sbert @xcite and universal sentence encoder @xcite for this challenge. our system (see figure 1 ) first splits the article text into a list of sentences including the title. then we compute embeddings with sbert and universal sentence encoder for all sentences and create the respective similarity matrices (similarity of all sentences of one article to all sentences of the other) of all article pairs. in the next step, we apply two different scoring approaches to these matrices: first, we apply simple maximum and average operations to the two matrices for four similarity scores, as proposed by @xcite . second, we feed the matrices into our simcnn to increase score accuracy (see appendix c). finally, we combine the five scores into a final score with a random forest regressor to get a stable prediction. optionally, an additional score for the publishing date distance of both articles can be computed and merged with the random forest result to refine the prediction. ## experimental setup after outlining the model components and their interactions in the previous section, here we cover the data sets, experiments and training processes used to configure the separate parts of the system. the sole evaluation metric for this competition is the pearson correlation coefficient r @xcite , which describes the linear association between two related variables x and y . a score close to -1 or 1 implies that a linear equation can express the relationship between the two variables almost perfectly, while a score of 0 indicates no correlation. ## results our system ranked 9th place in the competition with a pearson score of 0.759. to find the best models for submission, we tested the three last stages of our system on the aforementioned validation set separately: just the simcnn score, the prediction of the random forest regressor and the weighted average of the random forest score and the publishing date score. the results in the language distribution in appendix b shows a shift from latin languages and article pairs to vastly different and more complex languages: chinese-chinese article pairs account for over 15% of article pairs in the evaluation data. russian, polish and arabic articles also occur often, frequently in combination with other article languages such as english, french and german. this leads to problems in the preparation of the article texts for our system because our sentence tokenizer only supports english or similar texts and many of the new languages have a different structure and alphabet. when taking a look at the performance per language pair of our model in all things considered, we achieved satisfying results with just slightly modified publicly available models to create sentence embeddings and a cnn to work with them. the system only depends on an article's text, title and sometimes the publishing date, if it is available, and is still able to achieve a pearson correlation score of about 0.6 even for the most difficult examined language. ## conclusion pair-wise sentence comparison is a simple way to calculate the similarity of texts of arbitrary length if suitable multilingual models for sentence embeddings are available. with simple matrix operations like taking the maximum or average and a random forest regression algorithm, good results can be achieved. after introducing the more complex simcnn and combining it with a score of the publishing dates of the article pairs, our model surpassed a pearson correlation coefficient of 0.8 in some conditions. nevertheless, the current state of the system leaves many things to be improved: the sentence tokenizing currently only reliably works for english and similar languages. with a sentence splitting algorithm, that is capable of differently structured languages like chinese or japanese, results on such articles could be greatly improved. further, our pair-wise sentence similarity matrix approach could be extended to named entities like locations and persons, which we think would also greatly improve accuracy, but for that, a more sophisticated multilingual algorithm for named entity recognition would be needed.