BL . R esearch at S em E val-2022 Task 8: Using various Semantic Information to evaluate document-level Semantic Textual Similarity (2022)

Abstract: This paper presents our system for document-level semantic textual similarity (STS) evaluation at SemEval-2022 Task 8: “Multilingual News Article Similarity”. The semantic information used is obtained by using different semantic models ranging from the extraction of key terms and named entities to the document classification and obtaining similarity from automatic summarization of documents. All these semantic information’s are then used as features to feed a supervised system in order to evaluate the degree of similarity of a pair of documents. We obtained a Pearson correlation score of 0.706 compared to the best score of 0.818 from teams that participated in this task.

## introduction measuring semantic textual similarity has been a research subject in natural language processing, information retrieval and artificial intelligence for many years. accurate modelling of textual similarity is fundamental for many applications. previous efforts have focused on comparing a short text with a long text (e.g., web search), two sentences or other short text sequences (e.g., paraphrase recognition, image retrieval by captions and twitter tweets search). there are many other tasks requiring computing the semantic similarity between two long texts (e.g., document classification, document clustering and tracking the similarity of news coverage between different regions). semantic textual similarity (sts) measures the level of semantic equivalence between two textual contents. in this paper, we are interested to develop a system that identifies news articles that provide similar semantic information. more specifically, given two news articles, we aim to compute their similarity based on four characteristics: geolocation, time, shared entities, and shared narratives. to achieve this goal, we must identify important elements of the news articles content, such as the discussed event, location, time, and people involved. more precisely, the task 8 of semeval-2022 edition focuses on the analysis of documents (long texts) that can share different semantic information and can be from different natural languages in both monolingual and crosslingual settings. the sts task has been held in semeval since 2012. more precisely, the semantic similarity systems between sentences using paraphrase datasets have been proposed @xcite . in the 2013 edition of joint conference on lexical and computational semantics -*sem @xcite , sts task was focused in using cultural heritage items which are described with metadata such as title, author, or description. thereafter, systems that compare snippets of text are proposed in the 2016 edition of semeval @xcite . afterwards, the 2017 edition of semeval proposed to compare sentences in a dimension of monolingual and cross-lingual contents @xcite . in the last few years, many semantic bl.research at semeval-2022 task 8: using various semantic information to evaluate document-level semantic textual similarity sébastien dufour † , mohamed mehdi kandi * , karim boutamine † , camille gosset * , mokhtar boumedyen billami * , christophe bortolaso * , youssef miloudi †* † carl berger-levrault, 361 all. des noisetiers, 69760 limonest, france * berger-levrault, 64 rue jean rostand, 31670 labège, france {sebastien.dufour, youssef.miloudi}@carl.eu, {mohamed.kandi, karim.boutamine, camille.gosset, mb.billami, christophe.bortolaso}@berger-levrault.com similarity datasets and systems have been explored. among the elements that characterize this new edition of semeval-2022 compared to 2017 is the fact that we are interested in measuring the similarity between long news contents and not sentences, snippets of texts or short texts. we encountered many challenges: first, the content scraping of the provided url (in the first attempt, we had empty or incomplete content in many examples). then, we have many languages (new languages in the test set are not present in the train set). finally, it was challenging to choose the type of the predicted score: decimal or integer, which is different according to the number of human annotators per document. in this paper, we present our approach for semeval-2022 task 8. the system that we propose is based on the computation of different scores by document pairs. these scores are used as features to train a supervised system. regarding the multiple languages, we have developed a dedicated model for some of them, and translations in a pivot language for others. the paper is organized as follows: we describe the problem and the data in section 2. then, we give an overview of the related work in section 3. next, we present the proposed system in section 4. we detail the performed experiments and the results in sections 5 and 6. finally, we conclude the paper in section 7. ## related work semantic textual similarity deals with determining how similar two pieces of texts are. this can be done by assigning a rating from 1 to 4 (or 1 to 5) for more similar to less similar content pairs. related tasks are paraphrasing or duplication identification. there are many interesting works for sts, an evaluation toolkit for universal sentence representations, named senteval has been proposed @xcite . it includes 17 downstream tasks foot_0 , including common sts tasks from 2012-2016. we describe in this section some existing systems from the previous editions of semeval. @xcite proposed three featureengineered models using random forest, gradient boosting, and xgboost regression methods. their features are based on n-gram overlap; edit distance, longest common prefix/suffix/substring, tree kernels, word alignments, to cite a few. they also propose four deep learning methods. the difference between the methods is the approach to sentence embeddings using either: averaged word embeddings, projected word embeddings, a deep averaging network, or lstm (long-short term memory) @xcite . to build the global model, they average scores of the deep learning and the feature-engineered models. @xcite use sentence information content with wordnet @xcite and bnc word frequencies @xcite . one variant uses sentence information content exclusively. another variant uses ensembles information content with @xcite 's alignment method. the third variant uses ensembles information content with a cosine similarity of summed word embeddings with an idf (inverse document frequency) weighting scheme @xcite . @xcite proposed a convolutional deep structured semantic model for the generation of sentence embeddings. the embeddings are compared using cosine similarity and element-wise difference with the resulting values fed to additional layers. this architecture is similar to @xcite 's deep learning models. @xcite proposed a feature engineering approach that they complete with deep learning. ensembled components include alignment similarity; string similarity measures such as matching n-grams, summarization, machine translation (mt) metrics, an rnn (recurrent neural networks), and rcnn (recurrent convolutional neural networks) over word alignments, and a bilstm (bidirectional long-short term memory) networks. ## system overview some strategies can be considered to resolve the task 8 of semeval-2022 : the advantage of the first and the third strategies is that they avoid translation to a pivot language. the second strategy has the advantage of enabling consistent training and being able to handle all languages using a single model. our system is neither based on the translation of all texts in english, nor on the construction of a multilingual model. we have chosen to build learning models in some main languages, namely: english, french, spanish, german, arabic and turkish. we abandoned polish language because we did not find adequate pretrained models. when two texts to be compared are not in the same language, they are translated into the main language selected. a fundamental point for the final score to be generated is the choice on the precision of the answer in terms of decimals. we have noticed that for some languages, the overall score obtained is an integer. in english, for example, there is a decimal score and more annotators. the actual number of annotations cannot be determined in advance. we have therefore defined rules to complete our evaluation according to the score obtained and the language. we did not use the metadata of the news articles (authors, dates, newspaper, tags, etc.) because these data were incomplete. we simply retained both titles and text contents. we also noticed: the models are pretrained on different summarization corpus variants, for example mlsum, the multilingual summarization corpus @xcite . 3. we used identification and extraction of keywords/'key terms' in titles and content texts by using the pke toolkit library @xcite . the extracted tags are nouns, proper nouns, verbs, and adjectives only with stemming adding the ten semantically closest words by using word2vec @xcite with gensim library @xcite , if a model exists in a specific language. we compute the number of common terms in both texts. 4. we used identification and extraction of common named entities between titles and content texts, namely: places, persons, organizations, and dates. we used two libraries: spacy @xcite and stanza @xcite , and pretrained transformers models. as for keywords and key terms, we compute the number of identical entities in both texts. 5. for the various geographical detected entities (cities, regions, countries), we calculate a score for the proximity between places based on geocoding foot_2 . 6. we used the zero-shot classification models @xcite based on press topics that we have defined manually: politics, sport, health, economy and technology. the similarity score obtained reflects the number of common topics between two texts. for our english model, the various elements found allowed to have a good performance quickly (approximately 0.85 of pearson correlation). the only concern was the strong imbalance of the training dataset that we needed to rebalance. the french model had a poor training dataset (only 72 pair examples). thus, we selected a more efficient ner (named entity recognition) transformer model foot_3 than spacy; and supplemented and balanced it to 200 pair examples with spanish train texts to obtain a correct performance. we also focused on optimising the turkish model in the same way with an efficient ner transformer foot_4 . we have not worked to optimize the german model which ought to have been much better. ## experimental setup we applied our scoring models to every pair of titles and content texts. for polish language and the new languages observed in the evaluation dataset like chinese, russian and italian, all texts were translated into english with deep translator library foot_5 (google translate model) and then applied the english model. when there are two different languages, they are translated into english (for de_en, sp_en, pl_en and zh_en pairs), french (for fr_pl pairs), spanish (for sp_it pairs) or german (for de_pl and de_fr pairs). finally, a compromise strategy was used between the score obtained in classification (integers) and in regression (value between 1 and 4 with selected rounding). ## results and analysis for the translation part, we observe a fast drift according to the languages: the zh_en examples translated into english remain very accurate (0.79) but the zh_zh examples only obtain an average score of 0.69. we thus have a significant performance reduction in it_it (0.74) and sp_it (0.61) compared to cases when english is used. we found that most of the large deviations in evaluation (45 greater than 2 and 98 greater than 1.5) were related to scraping errors (blank or inconsistent text) where 2 and 1.5 are "overall scores". this derivation also resulted in an evaluation of 3 to 4 on our part for an actual close to 1. we did not find a reverse case where our score was close to 1 and the actual close to 4. we observe in conclusion that 71% scores of pairs were excellent (below 0.1), 84% were good (below 0.5) and 96% below 1. we believe that our english, french, turkish and spanish models are correct and could have been further optimized cleanly. we had technical difficulties in making a correct arabic model. as for the german model, we did not work on it enough and it should have obtained a performance close to 0.80. ## conclusion in this paper, we described our supervised semantic textual similarity system developed for the semeval-2022 task 8 and the result of the corresponding run we submitted. our system uses different features reflecting the similarity that can be obtained, for example, between shared key terms and named entities, or even topics by using zero-shot learning for text classification systems. in addition, we use geolocation for location entities and measure the semantic similarity through the use of lexical embeddings at the sentence level (text title) and paragraph level (text summarizer obtained automatically by using transformers). beyond the use of a supervised system to measure the degree of similarity between two given texts, we are in a context of documents that can come from different languages by processing both pairs of monolingual documents and pairs of cross-lingual documents. since the test corpus may contain documents written in natural languages not processed during learning phase, our system is able to perform an automatic translation into a pivot language in order to project new documents into already known spaces.