SATL ab at S em E val-2024 Task 1: A Fully Instance-Specific Approach for Semantic Textual Relatedness Prediction (2024)

Abstract: This paper presents the SATLab participation in SemEval 2024 Task 1 on Semantic Textual Relatedness. The proposed system predicts semantic relatedness by means of the Euclidean distance between the character ngram frequencies in the two sentences to evaluate. It employs no external resources, nor information from other instances present in the material. The system performs well, coming first in five of the twelve languages. However, there is little difference between the best systems.

## introduction semantic similarity between words, phrases and texts has long attracted the attention of nlp researchers. it is obviously a useful source of information in tasks such as information retrieval, text summarization, question answering or machine translation @xcite . it has been the subject of several shared tasks within semeval since 2012 @xcite @xcite . more recently, interest has also focused on semantic textual relatedness (str), which is supposed to be a more general concept. as @xcite point out, two sentences must be paraphrases or present an entailment relation to be semantically similar, whereas to be related, it is sufficient that they deal with similar themes or express similar points of view on a given issue. work on str is less advanced due to the lack of annotated datasets on this dimension @xcite . it should be noted, however, that the human annotators who evaluated semantic textual similarity for the silk dataset @xcite clearly evaluated relatedness, since they considered pairs of sentences that contradict each other as semantically very similar (96% similarity), such as in silk instance 466: -a man is performing a trick on a green bicycle. the silk dataset contains many other examples of this kind of judgement. this observation suggests that the term "relatedness" is more appropriate to describe this field of research, at least when dealing with the intuition of native speakers. it also suggests that techniques that are effective in automatically estimating semantic similarity should also be effective in estimating relatedness. these are mainly state-of-the-art deep learning algorithms @xcite . in this context, ousidhoum et al. ( 2024b ) have proposed the semeval 2024 task 1, which has a number of specific features compared with previous work. firstly, the task focuses on relatedness, and is based on material consisting of sentence pairs that have been annotated on this dimension by native speakers. secondly, the task is highly multilingual, covering more than ten languages, some of which are very poorly resourced. finally, it includes three subtasks: supervised, unsupervised and crosslingual. in the supervised subtask, the systems were to be trained using training datasets provided by the task organizers. in the unsupervised subtask, no datasets labeled according to semantic relatedness or semantic similarity could be used. in the crosslingual subtask, the system had to be trained on a language other than the target language. ## the proposed approach due to its highly multilingual nature (twelve languages), the unsupervised subtask seemed a priori to be particularly interesting for the development of a generic approach, as language-independent as possible. this would be the case of a system that estimates the semantic relatedness of a pair of sentences without recourse to any resources external to the material and even without taking into account the other instances present in the material. a system takes other instances into account when, for example, it weights an instance features according to their frequency in the complete material, using the classic tf-idf. a system is completely independent of other instances when the processing of one instance is not affected in any way by the other instances it has to predict. the system proposed by the satlab fulfills this requirement by using the euclidean distance between the two sentences, calculated on the basis of the frequency of the ngrams of characters that make them up. if such a system proves successful to predict semantic relatedness, it could become a potential candidate for the analysis of any language. admittedly, such a system is more akin to a baseline than a state-of-the-art system. however, it should also be noted that systems based on character ngrams have for many years been considered particularly effective for nlp tasks such as language identification, error correction, information retrieval and even for hate speech and offensive content identification @xcite . character ngrams have the advantage of not requiring material to be tokenized, which can be problematic in some asian languages, and of being able to extract morphological information at very low cost @xcite . this paper presents satlab's participation in semeval 2024 task 1 with this fully instancespecific system. the following section introduces the task and describes the proposed system. the results obtained are then reported. ## the satlab system a single system was used for all twelve languages. it is adapted from the one developed for the authorship identification of source code @xcite . this system takes as input each pair of utterances and outputs a distance between them without any other information, either from the rest of the material or external to it. each pair of utterances is therefore processed in a way that is completely independent of the other pairs present in the material. the only pre-processing is the lower-casing of all texts as included in sas. i have to admit that it's not obvious to me what impact this has on languages as unknown to me as kinyarwanda or amharic. no tokenization or lemmatization has been applied. the system uses character ngrams made up of 1 to 5 characters. all characters are taken into account, including spaces, punctuation marks, symbols, characters from other writing systems, etc. the ngrams at the beginning and end of each statement are distinguished from the others. all ngrams in a statement are retained, so there is no frequency threshold. the frequency of each feature is weighted by a logarithmic function using the formula: 1 + log(f req). finally, the features of each statement are weighted by the l2 norm (thus instance-wise). most of these system components have be taken from the one developed for a difficult language identification problem @xcite . the euclidean distance between the sets of ngrams of each utterance in a pair is used to estimate the semantic dissimilarity between these utterances. before submission, these distances are transformed into similarity by ranking them from largest to smallest. no information is lost through such ranking, since the organizers have chosen a rank correlation as the efficiency criterion. ## ethical considerations the ethical issues raised by this research are identical to those described by the researchers who collected the data @xcite and by the researchers who organized this task @xcite .