Team TMA at S em E val-2022 Task 8: Lightweight and Language-Agnostic News Similarity Classifier (2022)

Abstract: We present our contribution to the SemEval 22 Share Task 8: Multilingual news article similarity. The approach is lightweight and language-agnostic, it is based on the computation of several lexicographic and embedding-based features, and the use of a simple ML approach: random forests. In a notable departure from the task formulation, which is a ranking task, we tackled this task as a classification one. We present a detailed analysis of the behaviour of our system under different settings.

## introduction detecting similar news is a key component of media monitoring, such as the one done by the europe media monitor 1 , which collects daily about half a million articles in more than 80 languages. this shared task @xcite allows to study two important challenges that arise in practice: articles can be similar to several extent and in different ways; massive multilingualism requires language agnostic approaches. we report in this paper the result of an experimental approach to tackle this task. we considered computing simple lexicographic and embedding-based features and using simple ml approach for complexity reasons, having in mind that pair-wise comparison of half a million articles each day is not possible with heavier solution without massive resource cost. there are two important tasks when building such large scale news clustering: determining which pair of articles are worth comparing, and computing the actual similarity between pairs of articles. while this shared task deals only with the second approach, the presented system has been designed to tackle also the first one. for each article we consider separately the similarities related to 4 fields of the news item: the title, the description, the first sentence and the snippet. the snippet is constituted of the first 4 sentences comprised within the first 512 characters of the text -everything after was ignored, sentences were not truncated. we present here the set of features that have been computed for each of these fields, and for some cross field (title-description, title-first sentence, description-first sentences), all features are normalized. lexicographic measures: we consider two sets of representation: the set of words, and the set of multi-word expressions (mwe). the mwe were computed by splitting the text around stopwords and punctuation marks, essentially similar to rake keyword extraction @xcite . the features include the proportion of matching words and proportion of matching mwe between both articles. in the case of mwe, a margin of error was allowed in that two non strictly equal mwe were considered equal if the longest common subsequence between them was of 75% the length of the longest one. for both words and mwe we compute the raw count of elements in common and the length of the corresponding span of text. embedding-based measures: all embeddingbased measures are based on laser embeddings @xcite which are aligned multilingual embeddings covering over 100 languages and that perform well for multilingual semantic comparison. they are bilstm based, and as such are much faster than transformer-based solutions, it also has to be noted that out of the box bert solutions do not perform well on the task of semantic similarity, and laser embeddings have a better performance @xcite . on top of the pairwise comparison between the aforementioned fields, we also compute the equivalent of word mover distance @xcite between the first 4 sentences of the articles, by binning sentences in pairs of decreasing similarity. non linguistic features: the only non linguistic feature considered is the difference in publication date of the two articles. when developing the system, the results were checked for near misses, by considering all the clusters produced, and checking for missing links between elements of the clusters. this approach was able to catch such misses, including in the training data itself, but it was not used in the final system due to lack of time. ## data the training data provided by the organizers is a list of 4964 article pairs covering 9431 articles in 16 languages, while the 7 main languages represent 99% of the dataset. the pairs are associated with several similarity scores; one of them giving the overall similarity is ranked between 1.0 (maximum) and 4.0 (minimum) and is the only one considered in our approach. test data contains 4890 pairs over 9715 articles in 10 languages. there is a notable difference between both datasets, in that train data contained almost no pairs of article in different language, while such pairs represented 15% of the evaluation dataset. a major difference was also the introduction of chinese in the test set. ## discussion despite the problem of the shared task being posed as a regression one, we have chosen to address it as a classification problem. this has two direct negative consequences on the performance of the model: firstly due the mandatory discretization step of the real-valued evaluation score provided in the training data and expected by the evaluation system; secondly due to the fact that the notion of related classes, such as "similar" and "close", are lost and penalized as much as if "similar" had been predicted as "unrelated". this is clearly shown by the fact that the multilabel classifier trained on 4 labels performs significantly worse than a binary classifier trained on two classes, which has actually acceptable performances. despite being language agnostic our approach performs better in a same language setting than in a cross language one. this indicates either the high importance of the lexicographic as being a good predictor, or that multilingual embeddings perform significantly differently on different language pairs. the lexicographic-based features perform surprisingly well, only two points under the performance of the embeddings-based features when evaluated on the full training dataset contain. a potential reason for that could be that named entities can differ only slightly between languages and that the soft lexicographic measure used is good at capturing these variations. the performance of our approach is lower on the test dataset, such a drop in performance could be due to different reasons among others: the fuzzy lexicographic matching having lower performance in a cross language setting (much more prevalent in the test dataset than in the training); the difficulty to download data and correctly extract it, making the data incomplete; the heuristic reconstitution of missing description and text content could have introduced noise in some cases; maybe the fact that only tiny snippets of the full article were considered; or potentially a more fundamental limitation of the approach. our approach could be improved by considering the actual language pairs as features, using more advanced features based on named entity extraction, and using exclusively data without missing values. however, we lacked time to investigates all of these configurations. interestingly, of the embeddingbased features, the title and the description are the most important features, and work better than more advanced ones, such as word mover distance applied to the sentences of the snippets. given these results, we can argue that the language agnostic approach we developed is an interesting solution for a coarse grain similarity evaluation, but not for a fine grained one. given the fast computation time, a few minutes to compute all the features on a cpu machine without using multi-threading, our approach could be used as a preprocessing step before using more precise but also more time consuming approaches. particularly, this approach is interesting when a news processing system has to process hundreds of thousands articles a day, preventing the use of costly solutions. ## conclusion in this paper we presented the system we used to tackle the multilingual news clustering shared task at semeval-22. our approach is language-agnostic and inherently multilingual. it relies on a set of relatively simple lexicographic-and embedding-based features, and as such is able to process documents efficiently. we tackle the task as a classification problem rather than a regression problem. our approach performs satisfyingly when evaluated over a simpler binary classification problem.