F ired_from_ NLP at S em E val-2024 Task 1: Towards Developing Semantic Textual Relatedness Predictor - A Transformer-based Approach (2024)

Abstract: Predicting semantic textual relatedness (STR) is one of the most challenging tasks in the field of natural language processing. Semantic relatedness prediction has real-life practical applications while developing search engines and modern text generation systems. A shared task on semantic textual relatedness has been organized by SemEval 2024, where the organizer has proposed a dataset on semantic textual relatedness in the English language under Shared Task 1 (Track A3). In this work, we have developed models to predict semantic textual relatedness between pairs of English sentences by training and evaluating various transformer-based model architectures, deep learning, and machine learning methods using the shared dataset. Moreover, we have utilized existing semantic textual relatedness datasets such as the stsb multilingual benchmark dataset, the SemEval 2014 Task 1 dataset, and the SemEval 2015 Task 2 dataset. Our findings show that in the SemEval 2024 Shared Task 1 (Track A3), the fine-tuned-STS-BERT model performed the best, scoring 0.8103 on the test set and placing 25th out of all participants.

## introduction nowadays, there have been notable advancements in understanding and measuring pairwise semantic relatedness between texts within the domain of natural language processing. predicting semantic relatedness plays a significant role in improving search engines, question-answering systems, text summarization tools, and machine translation. however, previous works in natural language processing have mainly dealt with semantic similarity, a smaller aspect of relatedness, mainly due to the limited availability of relatedness datasets. besides, dealing with ambiguous words or phrases that have multiple meanings can make semantic relatedness difficult. understanding cultural context in language has been complex, and existing models have struggled to capture these variations. as language evolves, models struggle to adapt quickly to new linguistic patterns and expressions. to bridge these gaps, we need improved models that understand not just words but also context, cultural differences, and how language changes over time. semantic relatedness models have been developed using various transformer-based, deep learning, and machine learning techniques. traditional machine learning methods @xcite have relied on predefined rules and features and offered moderate results. these approaches have often struggled with complex semantic relationships. deep learning-based @xcite approaches have surpassed traditional machine learning models in capturing complex relationships, particularly in tasks requiring a deep understanding of context. however, transformer-based approaches @xcite have outperformed others when it comes to capturing semantic relationships, particularly in understanding context, managing long-range dependencies, and handling contextual embeddings. semeval has arranged a shared task named se-meval 2024 task 1: semantic textual relatedness (str) @xcite , introducing a novel dataset called shared task 1 (track a3) @xcite for determining the level of pairwise semantic relatedness between sentences based on the similarity score that ranges from 0.0 to 1.0. the primary goal of this task is to build a robust and accurate model to predict the semantic relatedness between pairs of english sentences. to accomplish this goal, we have used a variety of models, incorporating machine learning models (linear regression, random forest, xg-boost), models of deep learning (lstm, bil-stm), and pre-trained models based on transformer (roberta, bert-base-uncased). we have named our approach of using the bert-base-uncased model as sts-bert. by training and assessing every model, we have carried out a comparison analysis on the semeval 2024 task 1 (track a3) dataset @xcite , stsb multilingual dataset (may, 2021), semeval 2014 task 2 dataset @xcite and dataset provided for task 1 in semeval 2015 @xcite and have finally come to a conclusion that the sts-bert model has demonstrated better performance compared to others boasting an impressive spearman correlation coefficient of 0.81033 on the test dataset. the github repository that follows has the implementation details available - https://github.com/fired-from-nlp/ semeval-2024-task-1-track-a-eng . ## related works the associated works on semantic textual similarity can be generally categorized into three parts, approaches focused on machine learning, deep learning, and attention-based mechanism (transformer). among machine learning models, the support vector regression model has been applied for calculating the semantic relationship between two short sentences @xcite . in this system, three distinct measures, namely overlap in word n-gram, overlap in character n-gram, and semantic overlap, have been used for predicting similarity. in @xcite , a random forest-based approach has been utilized to find the semantic sentence similarity. the approach has relied on various similarity measures such as wordnet-based conceptual similarity, ic-based similarity, syntactic dependencies, and information retrieval-based similarity. traditional deep learning methods have depended on single or multiple granularity representations for detecting similarity. apart from that, a different architecture that has focused on multiple positional sentence representations has been proposed @xcite . it has used bi-lstm for generating representations that enable the model to capture better context understanding. another architecture has introduced a siamese adaptation of lstm @xcite . using a fixed-sized vector and a simple manhattan metric, the model transforms sentence representation that represents semantic relationships. another paper has described an architecture that has been built using deep learning paradigms @xcite . this architecture has been trained using a combination of features like features based on a string, features based on a corpus, and features based on syntactic similarity, as well as newer matrices derived from distributed word embedding. transformer-based approaches have surpassed both machine learning and deep learning models in calculating semantic sentence relationships. unlabeled text can be used to pre-train deep bidirectional representations using bert @xcite . bert can be fine-tuned to do various nlprelated tasks like semantic analysis. a replication of bert called roberta @xcite , has focused on hyperparameters and training data size to improve model performance. in this shared task, we have used bert-based pre-trained models as they have been proven to be superior to other models available. ## dataset we have employed the dataset made available as part of shared task 1 (track a3) of the semeval 2024: semantic textual relatedness (str) which contains 5500 samples in the training dataset and 250 samples in the dev dataset. besides, the stsb multilingual benchmark dataset (may, 2021), the semeval 2014 task 1 dataset @xcite and the semeval 2015 task 2 dataset @xcite have been used. ## system overview in this section, we have outlined our methodology to develop models for determining sentence relatedness. first, we have used various extraction strategies to extract characteristics and then utilized a variety of machine learning and deep learning algorithms. moreover, we have employed different transformer models to develop the system. figure 3 provides a summary of our working methods. ## experimental setup this section gives a summary of our experimental setup while training and evaluating our model architectures for semantic textual relatedness. ## experimental results in this section, we have showcased the experimental findings obtained during the training and evaluation stages of the proposed model for semantic textual relatedness prediction. among the machine learning models, we have found that the xgboost model with word2vec embedding has achieved the highest score of 0.3178. in the deep learning category, we have seen better performance as both lstm and bilstm models have higher scores than the machine learning models. the bilstm model achieved a score of 0.499, slightly outperforming the lstm model, which obtained a score of 0.445. in some cases, fasttext word embedding has obtained the best results compared to word2vec @xcite . therefore, we have also tested the performance of the model using fasttext embedding. however, the transformer-based models have clearly outperformed other models based on machine learning and deep learning. for instance, the roberta model achieved a score of 0.749, while our proposed sts-bert model demonstrated exceptional performance with an impressive score of 0.810. ## error analysis in the development phase external datasets, se-meval 2014 task 1 @xcite , semeval 2015 task 2 @xcite and multilingual benchmark dataset (may, 2021) along with the competition dataset have been utilized. hence the training set becomes more diverse and our model fails to learn about the relatedness between the sentences. the similarity scores of the external datasets ranged between 0.0 to 5.0. to make all the scores similar we have multiplied the scores of the competition dataset by 5.0 and normalized the whole training set by dividing all the scores by 5.0. due to multiple conversions of the range of scores, precision loss has occurred. sentence transformation has been another key reason for the poor performance of the model. when the second sentence is the transformation of the first sentence, our model can not detect it. for example, if the first sentence is in simple form and the second sentence is in the complex form of the first sentence, the model shows poor performance in that case. as a result, the overall performance of our proposed system has degraded. ## conclusion in this research, we have conducted a comparative performance analysis, assessing a range of machine learning, deep learning, and transformerbased models to predict the semantic textual relatedness between pairs of english sentences. we have utilized the task 1 (track a3) dataset provided in the shared task, along with additional external datasets, for training various models. our results indicate that the sts-bert model has outper-formed all other models, achieving an impressive score of 0.810. however, after analyzing errors, we have discovered that the slight score decrease is due to the integration of large external sts datasets with varying output ranges. to address this in future work, we plan to implement alternative strategies. moreover, we will work on task 1 (track b and c) to have more comprehensive findings.