Team dina at S em E val-2022 Task 8: Pre-trained Language Models as Baselines for Semantic Similarity (2022)

Abstract: This paper describes the participation of the team “dina” in the Multilingual News Similarity task at SemEval 2022. To build our system for the task, we experimented with several multilingual language models which were originally pre-trained for semantic similarity but were not further fine-tuned. We use these models in combination with state-of-the-art packages for machine translation and named entity recognition with the expectation of providing valuable input to the model. Our work assesses the applicability of such “pure” models to solve the multilingual semantic similarity task in the case of news articles. Our best model achieved a score of 0.511, but shows that there is room for improvement.

## introduction the multilingual news article similarity task 1 (semeval 2022 task 8) is designed as a shared task to encourage participants to build systems that check if a monolingual or cross-lingual pair of news articles belong to the same story @xcite . the task consists in providing a similarity score from 1 to 4 for a pair of news articles. in this paper, we describe the participation of the "dina" team in the shared task. in our participation, we took an exploratory approach focusing on language features, while trying several multilingual language models as baselines. these language models had been pre-trained for semantic similarity and are "pure" (without any fine-tuning conducted for the task), which we enhance to use jointly with state-of-the-art packages for machine translation and named entity recognition. our best-performing submission to the task is based on a "pure" paraphrase-xlm-r-multilingual-v1 model combined with the presence of overlapping named entities and overlapping dates, which 1 https://competitions.codalab.org/ competitions/33835 achieved a pearson's correlation score of 0.511 in the final evaluation. ## related work multiple prior semeval semantic similarity tasks @xcite @xcite @xcite focused on estimating the degree of semantic equivalence between two text fragments. in 2014 and 2015, two subtasks were proposed for semantic textual similarity in english and spanish @xcite . it was found that aligning words between sentences worked best for english, using features such as wordnet, word embeddings, or paraphrase databases. in 2016, a new crosslingual subtask was added for english and spanish. in 2017, participants were instructed to predict the degree of semantic similarity (namely, a continuous valued similarity score on a scale from 0 to 5) between monolingual and cross-lingual sentences in arabic, english and spanish @xcite . state-of-the-art deep learning models and feature engineered systems were implemented @xcite , and machine translation was widely used for cross-lingual and non-english setups, in order to convert two sentences into the same language. based on the corpus of english tasks data @xcite @xcite @xcite , the sts benchmark was presented for training and evaluation foot_0 . its extended version is often used to evaluate the performance of multilingual pre-trained models foot_1 . currently, state-of-the-art methods are based on pre-train transformer language models, which are fine-tuned for downstream tasks. multilingual pre-trained language models for 50+ languages are freely available 4 , such as distiluse-base-multilingual-cased models, paraphrase-multilingual-minilm-l12-v2, paraphrase-multilingual-mpnet-base-v2, labse @xcite , and other various models, such as paraphrase-xlm-r-multilingual-v1 foot_3 @xcite . all these models are applicable to the sentence similarity task. event clustering can be considered as a task that is close to the task of semantic similarity between news articles. @xcite used similarity metrics and ranking for clustering documents into monolingual and cross-lingual story clusters. linger and hajaiej (2020) used multilingual distil-bert and sentence-bert for multilingual document representation. in general, bert-like models with different averaging and pooling are widely used for document representation in the clustering task too. in the system proposed in our paper, we build on this line of research of leveraging large multilingual language models, which we further experiment with by incorporating machine translation and named entity recognition components. ## task and data the aim of the task was to check, at the document level, if a pair of news articles, which can be written in the same or different languages, provide similar information. the training set provided by the organisers includes 4,964 monolingual and cross-lingual pairs of news articles: 1800 english-english pairs, 857 german-german pairs, 577 german-english pairs, 570 spanish-spanish pairs, 465 turkish-turkish pairs, 349 polish-polish pairs, 274 arabic-arabic pairs, and 72 french-french pairs. for test data, the organisers added three new languages: italian, russian, chinese. for each entry in the dataset, different similarity scores are given for "geography", "entities", "time", "narrative", "style" and "tone", in addition to the main "overall" similarity score. these scores are based on a 4-point scale, from most (1) to least (4) similar. the aim of the task is to predict the "overall" score, but the participants can make use of the other auxiliary scores. for the final evaluation, the organisers proposed to use pearson's correlation between the predicted similarity score and the gold "overall" similarity score. correlation scores for different article pairs are then averaged to compute the final score. a script foot_4 provided by the task organisers enabled downloading news article pairs pertaining to train and test subsets. this script retrieves the urls of news articles, using the internet archive foot_5 to support this retrieval. ## data preprocessing upon downloading the news articles with the script provided by the organisers, we noticed that some texts were not correctly parsed: they contained only non-relevant texts (e.g. error messages), without the main content. therefore, we removed such problematic text pairs from the training set, based on the manually collected list of strings and heuristic rules for them, mostly for english (e.g. texts that started with "get full access to" or "take a free trial"); pairs with short texts (< 400 characters) that contained mostly metadata were also removed from the training set. finally, our training set consists of 4,228 text pairs. from texts both in the training and test sets, we also applied heuristic rules based on the manually collected list of strings (mostly for english) to remove non-relevant text fragments (e.g. "your browser does not support the audio element" or "share this item on twitter") from texts. all texts (in the training and test sets) were split into sentences using the stanza foot_6 python package, which supports all the languages under consideration. we relied on the assumption that news texts usually contain the most important information at the beginning, while the remainder of the story contains other less important details, according to the 'inverted pyramid' principle @xcite . according to this principle, the first paragraph or sentences of a news article are generally expected to cover the main information addressing the who, what, when, where, and why of a story. therefore, aiming to improve both the efficiency and effectiveness of our system, we only take the first 10 sentences of all non-english texts and we translate them into english using m2m100 models @xcite : 1.2b model foot_7 for the training set and 418m foot_8 model for the test set (the latter model was all the preprocessing, fine-tuning and evaluation tasks were performed using google colab and two nvidia geforce gtx 1080 ti graphics cards (22 gb ram). ## description of models and experimental setup in order to detect similarity at the document level, we focused on sentence-transformer models. all similarity scores were calculated based on cosine similarity. during the development stage, we conducted all our experiments on the training set, with a held-out subset reserved for evaluation. we tested different transformer models, including labse @xcite and paraphrase-xlmr-multilingual-v1 12 . based on the assumption that the main information about a story is expected to appear in the beginning of news articles, different setups for both models were taken: mean semantic similarity (cosine similarity) for all pairwise sentence-to-sentence similarities for the first 4 sentences in both articles (4 sent) and the first 5 sentences in both articles (5 sent); and the se-m2m100_418m 11 https://spacy.io/ 12 https://huggingface. co/sentence-transformers/ paraphrase-xlm-r-multilingual-v1 mantic similarity between the merged set of first 3 sentences from each article (3 sent merged). to come up with the best approach to submit to the shared task, we conducted three sets of experiments. experiment set 1. experiment set 2. in addition, we also conducted some baseline fine-tuning experiments. we took xlsum dataset as the biggest dataset with available news texts that contains most of the languages present in our train and test sets (arabic, chinese, english, spanish, russian, and turkish are included; it does not include german, italian, and polish). xlsum consists of 1.35 million articlesummary pairs from the bbc in 44 languages @xcite . we selected a smaller sample of 98,697 news texts with as balanced representation of 6 languages as possible (15,000 or fewer random examples for each language). unsupervised domain-specific fine-tuning of the paraphrasexlm-r-multilingual-v1 model on the news dataset (1 epoch with multiplenegativesrankingloss) did not improve the results: pearson's correlation score was 0.63 for all text pairs from the preprocessed training set, and 0.71 for only english-english pairs. masked language model approach (1 epoch, batch size 16) also yielded worse scores (0.41 and 0.51 respectively). ## conclusion this paper presents the participation results of our team "dina" in the multilingual news similarity shared task held as part of semeval 2022. we tested a range of state-of-the-art pre-trained multilingual transformer models, which were further tested by incorporating features based on dates, machine translation and named entity recognition. our best model achieved a pearson's correlation score of 0.511. it can be considered as a moderate performance score with substantial room for improvement, based on performance scores from other participants in the task, where the best system achieved a score of 0.818. in future experiments on cross-lingual semantic similarity of news texts, we aim to focus on more sophisticated fine-tuning techniques for domain adaptation on a further preprocessed and cleaned dataset.