NLP - LISAC at S em E val-2024 Task 1: Transformer-based approaches for Determining Semantic Textual Relatedness (2024)

Abstract: This paper presents our system and findings for SemEval 2024 Task 1 Track A Supervised Semantic Textual Relatedness. The main objective of this task was to detect the degree of semantic relatedness between pairs of sentences. Our submitted models (ranked 6/24 in Algerian Arabic, 7/25 in Spanish, 12/23 in Moroccan Arabic, and 13/36 in English) consist of various transformer-based models including MARBERT-V2, mDeBERTa-V3-Base, DarijaBERT, and DeBERTa-V3-Large, fine-tuned using different loss functions including Huber Loss, Mean Absolute Error, and Mean Squared Error.

## introduction semantic textual relatedness (str) is a natural language processing (nlp) task that focuses on measuring the degree of semantic relatedness between two pieces of text. unlike tasks such as semantic textual similarity (sts), which specifically assess the degree of similarity between texts, str considers a broader notion of relatedness, encompassing various types of semantic relationships between words, phrases, or sentences. the goal of str is to quantify how closely related two pieces of text are in terms of their underlying meaning or semantic content. this relatedness can encompass a wide range of semantic relationships, including: in the context of str, annotators or models are typically presented with pairs of text and asked to judge the degree of relatedness based on the presence of shared concepts or semantic associations. annotators might provide relatedness scores or labels indicating the strength of the relationship between text pairs. in this paper, we present our findings on se-meval 2024 task 1 track a: supervised semantic textual relatedness @xcite . our method consists of various transformer-based approaches @xcite fine-tuned using different loss functions including huber loss, mean absolute error, and mean squared error. the rest of the paper is structured in the following manner: section 2 provides the main objective of the task. section 3 describes our system. section 4 details the experiments. and finally, section 5 concludes this paper. ## task description this task aims to predict the semantic textual relatedness (str) of pairs of sentences across 14 different languages. participants will rank sentence pairs based on their semantic closeness, ranging from 0 (completely unrelated) to 1 (maximally related), as determined manually. teams can submit entries for one, two, or all of the following tracks: ## system description to tackle the semeval 2024 task 1 track a: supervised semantic textual relatedness, we fine-tuned several transformer-based models on an augmented training dataset and with different loss functions including huber loss, mean absolute error, and mean squared error. the different steps of our system are described as follows: in the context of semantic textual relatedness tasks, the choice of loss function plays a critical role in guiding the training process and optimizing model performance. given the diverse nature of textual data and the wide range of semantic relationships to be captured, employing a variety of loss functions can offer several advantages. firstly, the huber loss function provides robustness to outliers by combining the advantages of mean absolute error (mae) and mean squared error (mse). mae, which calculates the average absolute difference between predicted and target values, is less sensitive to outliers compared to mse, which squares the differences. by behaving like mse for large errors and like mae for small errors, huber loss ensures that the training process is less influenced by outliers, thereby enhancing the model's ability to generalize to unseen data. secondly, mean absolute error (mae) serves as a straightforward and intuitive loss function that penalizes deviations from the target scores equally, irrespective of their direction. in tasks such as semantic textual relatedness, where the goal is to predict similarity scores between sentence pairs, mae provides a direct measure of the magnitude of errors, facilitating easy interpretation and evaluation of model performance. lastly, mean squared error (mse) emphasizes the importance of accurately predicting similarity scores by penalizing larger errors more severely than smaller errors. in scenarios where precise estimation of the degree of relatedness between sentence pairs is crucial, mse can effectively guide the training process towards minimizing the squared differences between predicted and target values, thereby optimizing model performance. by leveraging a combination of these loss functions during the fine-tuning process, we aim to capitalize on their respective strengths and enhance the robustness and effectiveness of our models in capturing semantic relationships within textual data. this approach enables us to optimize model performance across various linguistic contexts and achieve competitive results in tasks requiring accurate assessment of semantic textual relatedness. the decision to augment the data was validated through fine-tuning the models on concatenated train and dev sets, as well as on concatenated train and dev sets with pair shifting. interestingly, our analysis revealed that pair shifting significantly enhanced the results on the development sets. ## experimental results we experimented our model on the semeval 2024 task 1: semantic textual relatedness (str) test set @xcite . the experiment has been conducted in kaggle environment 1 , the following libraries: transformers -hugging face 2 @xcite , and keras 3 were used to train and to assess the performance of our models. ## conclusion in conclusion, this paper has presented our system and findings for semeval 2024 task 1 track a: supervised semantic textual relatedness. the primary objective of this task was to detect the degree of semantic relatedness between pairs of sentences across multiple languages. our submitted models, leveraging various transformer-based architectures including marbert-v2, mdeberta-v3-base, darijabert, and deberta-v3-large, fine-tuned with different loss functions such as huber loss, mean absolute error, and mean squared error, achieved competitive rankings across different language tracks. our approach highlights the effectiveness of leveraging advanced transformer-based models and fine-tuning techniques to capture intricate semantic relationships within textual data. by incorporating diverse loss functions during the training process, we aimed to optimize the models' performance and enhance their robustness across various linguistic contexts. moving forward, further research in semantic textual relatedness should focus on refining existing methodologies, exploring novel architectures, and addressing cross-lingual challenges. additionally, efforts to incorporate additional linguistic features and develop more comprehensive evaluation metrics can contribute to advancing the state-of-the-art in this field. overall, our contributions underscore the significance of semantic textual relatedness in natural language processing tasks and pave the way for the development of more sophisticated and contextaware systems capable of understanding and interpreting textual data with greater precision and accuracy.