Sharif- STR at S em E val-2024 Task 1: Transformer as a Regression Model for Fine-Grained Scoring of Textual Semantic Relations (2024)

Abstract: This paper explores semantic textual relatedness (STR) using fine-tuning techniques on the RoBERTa transformer model, focusing on sentence-level STR within Track A (Supervised). The study evaluates the effectiveness of this approach across different languages, with promising results in English and Spanish but encountering challenges in Arabic.

## introduction str delineates the meaningful association between linguistic units, showcasing conceptual proximity within a shared semantic frame @xcite . for instance, "cup" and "coffee" are related in meaning, yet they are not synonymous @xcite . despite its crucial role in various nlp applications such as spelling correction, word sense disambiguation, plagiarism detection, opinion mining, and information retrieval @xcite @xcite , str has garnered less attention compared to semantic textual similarity (sts) due to a scarcity of available datasets. addressing this gap, @xcite , and @xcite contributed to the field by constructing the first sentence-level str datasets. in this paper, we endeavor to tackle the str problem within shared task 1 @xcite , track a, leveraging supervised data in english, spanish, and arabic languages provided by @xcite . additionally, we briefly explore track c and provide supplementary details in appendix b as a secondary objective. building upon the findings of @xcite , which underscore the superior performance of fine-tuning transformer models in supervised tasks, our proposed system captures the relationship among sentences by fine-tuning the roberta transformer @xcite . at the core of our system, we employ a pre-trained roberta model as a regression model and fine-tune it to generate a floating-point value for the input text. during the pre-training process of roberta, the emphasis is placed on tasks related to nlu. this involves exposing the model to a diverse range of linguistic contexts and training it to comprehend the nuances of language. furthermore, the integration of a classifier head enables sentence classification, a pivotal aspect of our system architecture elaborated upon in section 3. our experimental results showcase promising performance on english and spanish datasets, achieving respective correlation rates of 0.82 and 0.67 on test data, surpassing the baseline correlation set by semeval-2024 at subtask a @xcite . however, the model's performance on arabic data falls short, yielding only a 38% correlation on development data. we attribute this discrepancy to differences in the underlying roberta model and its training methodology across latin and non-latin languages, a topic further explored in section 5. to promote reproducibility and facilitate future research endeavors, the complete codebase of our project has been shared on github foot_0 . ## system overview in this section, we present a comprehensive overview of our system's architecture, outlining the key algorithms and modeling decisions that underpin our model. ## conclusion in our investigation, we focused on fine-tuning roberta for str, primarily targeting latin languages like english(0.82) and spanish(0.67). while our approach showed promising results for these languages, particularly in achieving high correlation, the outlook was less favorable for arabic(0.38). this echoes discussions in previous works, emphasizing the significant influence of the data on model performance. our exploration into track c, which is given in appendix b, further enriched our understanding of the challenges and opportunities in str system development. as a contribution to the field, we put forth several recommendations for enhancing str systems. firstly, we propose the development of additional transformer models trained on diverse language families, focusing on languages that share similarities with latin languages. furthermore, a comprehensive benchmark of models on the str dataset is essential, building on previous research that highlights the strong performance of models like albert, xlnet, and t5-11b on the sts dataset. moreover, the utilization of translation techniques and data augmentation methods could enhance model performance, particularly for languages with limited training data. in conclusion, our study sheds light on the nuances of str system development and underscores the importance of considering languagespecific factors and domain characteristics. by pursuing the avenues outlined in this paper, we aim to contribute to the advancement of str research and facilitate the development of more robust and accurate models for nlu tasks.