All-Mpnet at S em E val-2024 Task 1: Application of Mpnet for Evaluating Semantic Textual Relatedness (2024)

Abstract: In this study, we tackle the task of automatically discerning the level of semantic relatedness between pairs of sentences. Specifically, Task 1 at SemEval-2024 involves predicting the Semantic Textual Relatedness (STR) of sentence pairs. Participants are tasked with ranking sentence pairs based on their proximity in meaning, quantified by their degree of semantic relatedness, across 14 different languages. Each sentence pair is assigned manually determined relatedness scores ranging from 0 (indicating complete lack of relation) to 1 (denoting maximum relatedness). In our submitted approach on the official test set, focusing on Task 1 (a supervised task in English and Spanish), we achieve a Spearman rank correlation coefficient of 0.808 for the English language and 0.611 for the Spanish language.

## introduction the notion of semantic relatedness between language units has been foundational in understanding meaning. automatic determination of relatedness has wide-ranging applications, including assessing sentence representation methods, question answering, and summarization @xcite . two sentences are deemed semantically similar when they exhibit a relationship of entailment or paraphrases. in contrast, relatedness encompasses a broader concept, encompassing all commonalities between two sentences: whether they pertain to the same topic, convey the same viewpoint, stem from the same temporal context, one elaborates on the other, and so forth @xcite . historically, much of nlp research has focused on semantic similarity, predominantly in english. however, in the shared task 1 hosted at semeval 2024 @xcite , the organizers extend their coverage to include the following languages: afrikaans, algerian arabic, amharic, english, hausa, hindi, indonesian, kinyarwanda, marathi, moroccan arabic, modern standard arabic, punjabi, spanish, and telugu. with the advancement of machine and deep learning architectures in recent years, there has been a surge of interest in nlp. numerous efforts have been dedicated to creating algorithms capable of automatically identifying and categorizing text information available on the internet. in the literature, several strategies have already been proposed. in the last fifteen years, some of the most successful strategies have been based on svm @xcite , on convolutional neural network (cnn) @xcite , on graph neural network (gnn) @xcite , on ensemble models @xcite and, recently, on transformers @xcite . the increasing adoption of transformer-based architectures in academic research has also been bolstered by various methodologies showcased at semeval 2024. these methodologies tackle diverse tasks and yield noteworthy findings. for instance, at the task 2 @xcite , where to address the challenge of identifying the inference relation between a plain language statement and clinical trial reports is used t5 @xcite ; task 4 @xcite where is employed a mistral 7b model to detect persuasion techniques in memes @xcite ; and task 8 @xcite , that utilizes a distilbert model to identify machine-generated text @xcite . for our model development, we devised a twostage architecture. in the first stage, we utilized a sentence transformer specifically trained in a multilingual domain. subsequently, we computed the cosine similarity of the generated embeddings to predict the relatedness between the analyzed sentences. the remainder of this paper is structured as follows: section 2 provides background information on task 1 hosted at semeval-2024. section 3 presents an explanation of the submitted approach. we detail the experimental setup required to reproduce our work in section 4. the results of the formal assignment and pertinent discussions are presented in section 5. finally, we conclude with our findings and suggestions for future research in section 6. we make all the code publicly available and reusable on github foot_0 . ## background data for semantic textual relatedness (str) shared task 1 foot_1 includes sentence pairs labeled with scores representing the degree of semantic textual relatedness between them, ranging from 0 (completely unrelated) to 1 (maximally related). these scores have been determined through manual annotation using a comparative annotation approach to mitigate biases commonly associated with traditional rating scale methods. this annotation process ensures a high reliability of the relatedness rankings. the task involves predicting the str of sentence pairs in 14 different languages. participating teams were asked to submit systems for one, two, or all of the following tracks: ## system overview the illustration of the proposed approach is provided in the figure 1 . upon selecting a sample (i.e., a pair of sentences) from the dataset, the initial step involves encoding the first sentence using the all-mpnet embedding, thereby generating an embedding vector. subsequently, employing an identical procedure, the second sentence from the sample is also encoded. the resulting embedding vectors are then subjected to a cosine similarity computation, facilitating the derivation of the semantic similarity prediction between two sentences. to develop our model, we thought of a twostage architecture. in the first stage, we used a sentence transformer. this is a python framework for cutting-edge sentence, text, and image embeddings. the initial work is described in @xcite . more than 100 languages have sentences and text embeddings that can be computed using this method. sentences with a similar meaning can subsequently be found by comparing these embeddings, for example, using cosinesimilarity. semantic search, paraphrase mining, and semantic textual similarity can all benefit from these embeddings. the framework offers a huge selection of pre-trained models suited for different tasks and is built on pytorch and transformers. moreover, fine-tuning models is also feasible. the model used as sentence transformer is allmpnet-base-v2, and it is available on hugging-face foot_2 . the model is based on mpnet @xcite . mpnet introduces a novel pre-training approach that combines the strengths of bert and xlnet while addressing their respective limitations. unlike bert's masked language modeling (mlm), mpnet utilizes permuted language modeling (plm) to capture dependencies among predicted tokens more effectively. additionally, mp-net incorporates auxiliary position information as input, allowing the model to process full sentences and mitigate position discrepancy issues present in xlnet. pre-training of mpnet is conducted on a large-scale dataset exceeding 160 gb of text corpora, followed by fine-tuning on various downstream tasks such as glue @xcite and squad @xcite . experimental findings demonstrate that mpnet significantly outperforms both mlm and plm, achieving superior results across these tasks compared to previ-dataset all-mpnet cos_sim "the book" "a cat" embed_1 embed_2 0.19 figure 1 : block diagram of our proposed approach. given a sample from the dataset, the first sentence is encoded with an all-mpnet embedding for generating an embedding vector as output. also, the second sentence in the sample is encoded in the same way. then the two embedding vectors are compared using cosine similarity to produce the final prediction on the semantic similarity of the two sentences. the model was used to map all the words present in the text to a word embedding space. following the embeddings, the cosine distance between two sentences was calculated. the cosine similarity between the two embedding vectors is calculated as shown in the equation 1 . the value provided as cosine similarity was then provided as the requested prediction for the two sentences considered. our code is available online together with the predictions generated and sent in relation to the test set. the recent study by @xcite highlights that preprocessing for text classification tasks lacks significant impact when employing transformers. specifically, the study finds that the optimal preprocessing strategies do not substantially differ from performing no preprocessing at all, particularly in the case of transformers. consequently, in order to maintain our system's efficiency, speed, and computational lightness, we have opted to not conduct any preprocessing on the text. this decision aligns with the findings of the study and underscores the effectiveness of transformers in handling raw text data without the need for extensive preprocessing steps. ## experimental setup we implemented our model on google colab foot_3 . the library we used was sentence transformer. the library requires python foot_4 (>= 3.8) and pytorch foot_5 (>=1.11.0). the dataset provided for all the phases are available on the official competition page. on the basis of our preliminary experiments, we found beneficial to set the threshold value upon the cosine similarity equal to 0.5. we did not perform any additional fine-tuning on the mpnet embeddings. to run the experiment, a t4 gpu from google has been used. after the generation of the predictions, we exported the results on the json format required by the organizers. as already mentioned, all of our code is available on github. ## results the official evaluation metric for this task is the spearman rank correlation coefficient, which evaluates how closely the rankings predicted by the system align with human judgments. the evaluation script for this shared task is available on the github page, providing a standardized method for assessing the performance of participating systems. the formula to compute the spearman correlation coefficient is provided in the equation 2 . where d represents the pairwise distances of the ranks of the variables x i and y i , while n is the number of samples. in in the in parentheses is reported the position in the official ranking. even if our approach is simple and straightforward, here we want to qualitative analyze some results obtained with our approach to better motivate some classification mistakes. with regard to the english language, the first relevant misclassification sample is related to the sample pair: "this book is very compelling. -best book so far in the series!". our system predicts a cosine similarity equal to 0.47 while the actual similarity is 0.73. in this case, in fact, it is very hard to assess if "being very compelling" is so semantically similar to the concept of "being the best so far in a series". furthermore, looking at the sample: "a woman in a black coat eats dinner while her dog looks on. ## conclusion this paper introduces the utilization of an all-mpnet model embedding to tackle task 1 at semeval-2024. in our submission, we opted for a straightforward zero-shot learning approach, leveraging pre-trained transformers that are already tailored to a multilingual-domain. following this approach, we utilized the contextual embeddings generated by the sentence transformer, and we employed cosine distance to measure the similarity between pairs of sentences, thus quantifying the str between them. despite the effectiveness of our method, there remains room for improvement, as indicated by the final ranking. potential alternative approaches could involve leveraging the zeroshot capabilities of models such as gpt and t5, expanding the training data size by incorporating additional datasets, or exploring different methods of integrating ontology-based domain knowledge into our approach. furthermore, given the interesting results recently provided on a plethora of tasks, also few-shot learning @xcite @xcite or data augmentation strategies @xcite @xcite could be employed to improve the performance. compared to the best performing models, our simple approach exhibits some room for improvements. however, our qualitative analysis raised some concerns on the labels provided for the test set. then, we are not able to correctly assess the actual performance of our proposed approach. eventually, it is worth notice that thanks to our approach no further pre-training is required and the computational cost to address the task is manageable with the free online resources offered by google colab.