GIL - IIMAS UNAM at S em E val-2024 Task 1: SAND : An In Depth Analysis of Semantic Relatedness Using Regression and Similarity Characteristics (2024)

Abstract: The STR shared task aims at detecting the degree of semantic relatedness between sentence pairs in multiple languages. Semantic relatedness relies on elements such as topic similarity, point of view agreement, entailment, and even human intuition, making it a broader field than sentence similarity. The GIL-IIMAS UNAM team proposes a model based in the SAND characteristics composition (Sentence Transformers, AnglE Embeddings, N-grams, Sentence Length Difference coefficient) and classical regression algorithms. This model achieves a 0.83 Spearman Correlation score in the English test, and a 0.73 in the Spanish counterpart, finishing just above the SemEval baseline in English, and second place in Spanish.

## introduction the semantic textual relatedness (str) task @xcite aims at creating systems that measure str on pairs of sentences based on their closeness in meaning @xcite . this task is comprised of three tracks. tracks a and b focus on monolingual models. track a only accepts supervised models trained with the available tagged datasets, whereas track b focuses on the unsupervised approach relying on the same datasets but without the tagged score. track c is the cross-lingual case where the target language has to follow an unsupervised approach. the datasets provided consist of sentence pairs that were sampled from various semantic similarity datasets. this task expands upon classic sentence similarity comparisons, encouraging the use models and algorithms capable of analyzing more than mean-ing of a pair of sentences, focusing deeper on characteristics such as the syntactic structure of the sentences, lexicon relationships, as well as meaning and emotion. the gil-iimas unam team participated in track a. although it included nine languages we have only worked with the spanish and english dataset. track a is a regression problem since each dataset contains the sentence pairs as well as a corresponding sentence relatedness score that ranges from 0 to 1. the evaluation compares the ground values in the test set with the proposed model's prediction, meaning track a is evaluated using the spearman correlation. this paper makes use of the sand composition, a set of str relevant characteristics, as well as regression algorithms trained with these characteristics in order to predict the str score of other sentence pairs. in this paper the precise characteristics, algorithms, and parameters are presented as well as language based analysis. the final scores correspond to the best performing regression algorithm. our work also compares different metrics and their influence on the str task compared to classic semantic similarity, as well as the model's varying behavior over the different languages used. the paper is structured as follows: section 2 explains the theories and models that are the background of our proposal. section 3 explains the set of characteristics that have been chosen for our experiments. section 4 explains the configuration of the data and the experimental methodology. in section 5 we discuss the results obtained in the experiments, and compare them to the scores of other participants in the track. we close in sec-tion 6 with some conclusions and ideas for further experiments. ## related work in the field of str, various methodologies have been proposed. one such approach, outlined by @xcite , involves analyzing the relatedness between word bi-grams. they describe the construction of a dataset tailored for this purpose. to compute the str between bi-grams, or between bi-grams and unigrams, they utilize word embeddings represented as vectors generated by glove, fasttext, and models employing matrix factorization of word-context co-occurrence matrices. they explore various methods for composing bi-gram vectors, such as addition, multiplication, tensor product with convolution, and dilation. the relatedness between two vectors is determined by computing the cosine similarity between them. in a study by @xcite , the concept of semantic textual relatedness (str) is extended to encompass the comparison between entire sentences. the authors delineate the construction of a specialized dataset tailored for this task and show the annotation process applied to this dataset. their investigation delves into the influence of various linguistic factors, including lexical overlap, related words, related words belonging to the same part of speech, and the relatedness of subjects or objects, on the semantic relatedness between pairs of sentences. they represent each sentence as a vector and employ cosine similarity between these vectors as a metric for predicting semantic relatedness. to facilitate this analysis, they use static word embeddings such as word2vec, glove, and fasttext, as well as contextual word embeddings like bert and roberta. ## sand composition this section describes the sand (named based on the used characteristics: sentence transformers, angle embeddings, n-grams, sentence length difference coefficient) regression system used for the task. the str datasets are comprised of senbtence pairs and a target score, in order to train the model with task relevant information certain similarity metrics were chosen in order to create a vector of characteristics that represent each sentence pair, such vectors were used as training data for the regression algorithms. an initial approach relied on similarity metrics such as jac-card, and dice coefficients as well as jaro-winkler and levenshtein distance, nonetheless these metrics proved to be inefficient at training the model adequately, returning poor results when evaluated with partitioned training data. after revising the dataset and the nature of the sentences in question, the initial chosen characteristics were a coefficient analyzing the length of the sentences with and without stoptwords. consider x, y two sentences, then both coefficients are obtained from the following: an observation on lexical overlapping in highly related sentences led to the choice of an n-gram based coefficient. for @xmath0, 2, 3}, the n-gram coefficient is defined as: nonetheless standard encoder generated embeddings have shown to be improvable, for that the final characteristic relies on similarity based on angle-optimized embeddings @xcite . these type of embeddings optimize the cosine similarity saturation zones during training using complex space embeddings so that resulting vectors achieve a higher level of similarity. nonetheless the final comparison between angle vectors is done in the same manner as equation 3. once these characteristics are extracted from each sentence pair they are passed to various regression algorithms for training, validation and testing. for this task four regression algorithms were used. the reported scores correspond to the best model for each language. the precise details of the implementation are presented next. ## results and analysis the final sand composition was the result of an ablation test performed using various combinations of different characteristics. the previously mentioned regression algorithms were trained and tested using each individual characteristic as well as different combinations of each. the sand composition was the best performing combination for both english and spanish, each of the four regression algorithms achieved the best result in the ablation test with sand than with each simpler combination. tables 1 and 2 show the best spearman correlation for each characteristic combination as well as the model that achieved said score in each language when evaluating in the validation dataset. finally it is important to note that the embeddings created with pretrained models were the feature with the greatest impact on our model. even as an isolated measure they both prove to be better met- sand composition was able to achieve the best results of the ablation test meaning that regression models do benefit from the mix of characteristics and still be relevant in a competition setting. ## conclusion in this paper, we describe the sand composition for the str shared task, which is based on both semantic and lexical features, because we observe that: two sentences can share most of the words and apparently have no semantic relation but a high value of spearman coefficient and vice versa, they can share semantics without matching words. with this in mind the sand composition contains half semantic features, and half lexical ones. this approach allowed achieved the 18th place in english and 2nd place in spanish, with 0.83 and a 0.73 spearman correlation score respectively. in both cases the results are over the baseline and only 0.05 of the first place in english and 0.01 in spanish, meaning sand proves to have relevant characteristics. for future experiments added features that consider both semantic, lexical and contextual parts simultaneously might prove to be more efficient than various unrelated metrics. mixing word embeddings and post tagging might generate a relatedness score that proves to be more useful than separate similarity metrics.