S koltech NLP at S em E val-2022 Task 8: Multilingual News Article Similarity via Exploration of News Texts to Vector Representations (2022)

Abstract: This paper describes our contribution to SemEval 2022 Task 8: Multilingual News Article Similarity. The aim was to test completely different approaches and distinguish the best performing. That is why we’ve considered systems based on Transformer-based encoders, NER-based, and NLI-based methods (and their combination with SVO dependency triplets representation). The results prove that Transformer models produce the best scores. However, there is space for research and approaches that give not yet comparable but more interpretable results.

## introduction the semeval 2022 task 8 competition @xcite aims to develop systems that identify multilingual news articles that provide similar information. this is a document-level similarity task in the applied domain of news articles, rating them pairwise on a 4-point scale from most to least similar. the alikeness of news is measured in such a sense: how similar are them in geography, time, shared entities, and shared narratives. the developed approaches for solving the task can be applied to several different real-world tasks. the first one is the clustering of news. a lot of news-providing companies want thousands of articles from different publishers on one topic to be combined in a single page showing the news. another task is fake news detection. this task has become extremely important to solve lately. different articles on the same news story can be compared to find contradictions in facts and details. this can be an indicator that the story is fake like it is shown in @xcite . ## related work in @xcite , authors extract person, organization and location named entities (ne) and compose a vector for each of the category with the help of levenshtein distance function and tf-idf weighting function, which combines term frequency (tf) and inverse document frequency (idf). these 3 vectors are compared to 3 corresponding vectors of the second news with cosine distance. obtained scores are combined with the set of if-then rules. in @xcite the approach for cross-lingual transfer is proposed which is evaluated on the named entity recognition task. dialogue competition @xcite on russian news clustering has produced many promising methods which could be adopted to the multilingual case. most of them, like @xcite , are the variations of fine-tuning the transformer models and making ensembles. in @xcite , the authors developed the pipeline for checking the news on veracity. they compare embeddings of the news under consideration with ones from the database, using cosine distance. then they take the most similar news found and apply natural language inference (nli) model to obtain the probability that two texts contradict each other. this probability is used to decide whether the input news is fake. however, nli scores can be used to find the similarity between articles. ## methodology to solve the task we have tried several approaches. in the subsection 3.1 we will give an overview on methods exploiting pre-trained transformer models as the foundation. in the next subsection we will explain how the natural language inference (nli) problem can be reduced to the task of news article similarity. block 3.3 is dedicated to approaches based on the named entities extracted from the news texts. in the last section, the approaches which were tested to improve the quality of prediction during the post-evaluation period will be described. ## results our team has taken the 14th place among 32 participants. the best result for separate model, 0, 734 correlation, was reached by the transformeren-codercossim model. final results for all separate methods, as well as the best competition score, are provided in the reason for that could be that the nli model was trained on the xnli dataset, composed of short phrases. that is why it was decided to try to shorten the news with the extraction of svo triplets from them. the extracted triplets were joined to form a text which was forwarded to the input of the nli model. as you can see from when the other two distance scores correctly reflect the ground true similarity, like in the second example in ## conclusion we have tested several approaches, including two systems based on transformer-based encoders, two nli approaches (with fine-tuning and without), ner-based pipeline and the ensemble of these models. the best result was achieved by the ensemble of transformerencodercossim, trans-formerencodercls and fine-tuned nli models. to improve the scores the following things can be done. for models based on transformer-based encoders, sentence transformers can be tested. to improve the ner-based method additional features can be added (addition of nli features improved the correlation), and also we can apply the binary mask for the feature matrix not to take into account 0.5 values while calculating the loss during the training process can be applied. source code of our solutions is available online foot_6 . also, the hyper-parameters of the models can be found there.