S emantic CUETS ync at S em E val-2024 Task 1: Finetuning Sentence Transformer to Find Semantic Textual Relatedness (2024)

Abstract: Semantic textual relatedness is crucial to Natural Language Processing (NLP). Methodologies often exhibit superior performance in high-resource languages such as English compared to low-resource ones like Marathi, Telugu, and Spanish. This study leverages various machine learning (ML) approaches, including Support Vector Regression (SVR) and Random Forest, deep learning (DL) techniques such as Siamese Neural Networks, and transformer-based models such as MiniLM-L6-v2, Marathi-sbert, Telugu-sentence-bert-nli, and Roberta-bne-sentiment-analysis-es, to assess semantic relatedness across English, Marathi, Telugu, and Spanish. The developed transformer-based methods notably outperformed other models in determining semantic textual relatedness across these languages, achieving a Spearman correlation coefficient of 0.822 (for English), 0.870 (for Marathi), 0.820 (for Telugu), and 0.677 (for Spanish). These results led to our work attaining rankings of 22th (for English), 11th (for Marathi), 11th (for Telegu) and 14th (for Spanish), respectively.

## introduction semantic textual relatedness measures the conceptual and contextual similarity of two sentences. it specifies how alike the two sentences are in terms of meaning. determining semantic textual relatedness is crucial for various language-processing tasks, including contemporary technology, search engines, chatbots, virtual assistants, plagiarism detection, paraphrasing, question answering, text generation, and other related applications. it is possible to determine how comparable two natural language sentences are based on the quantity and quality of matched elements in each sentence. these matches offer essential insights into the relationship between and degree of semantic similarity between the two sentences and the likelihood of successful word matching in semantically equivalent text pairs. another critical aspect of semantic relatedness is understanding the context of sentences. singnificant research has been done in this area, specifically in semeval competition from 2012 to 2017 @xcite @xcite @xcite . most proposed methodologies perform better in high-resource languages like english and spanish but could be better in other low-resource languages like telegu, marathi, and bengali. in certain studies, translating from a low-resource language to english is conducted @xcite prior to semantic analysis, contributing to suboptimal performance. other factors include the failure of specific methods to capture the meaning of phrases and idioms within sentences effectively @xcite , as well as a tendency for some approaches to focus excessively on individual word meanings @xcite . this work addresses these shortcomings by proposing a transformer-based model, which gives us a good result in finding semantic textual relatedness. the main contributions of this work are: the code will be publicly available at https://github.com/ashrafulparan2/ semanticcuetsync-at-semeval-2024-task-1 . numerous studies have been accomplished on semantic relatedness in high-resource languages. hasan et al., 2020 presented the process for calculating semantic similarity and proposed a featurebased metric for building semantic vectors. their knowledge feature-based method found similarity measure of 0.82. abdalla et al., 2021 proposed a dataset to explore questions on what makes sentences semantically related. reimers and gurevych, 2019 suggested using a sentence transformer model, which creates a 768-dimensional dense vector space from sentences and paragraphs. the relatedness of two sentences can be assessed using this model. their approach achieved the best score of 0.8492 with sroberta-stsb-base model. three transformer-based clinical semantic textual similarity models intended to detect semantic relatedness in medical data were presented by yang et al., 2020. chen et al., 2022 proposed a semisupervised sentence embedding technique called gense, which effectively uses large-scale unlabeled data. it achieved promising results on several sts datasets with an average correlation score of 0.8519. meanwhile, gatto et al., 2023 evaluated sentence similarity among texts using large language models (llm). according to their research, chatgpt and other models are proficient in recognizing textual similarity within particular areas. in addition to the transformer models, verma and muralikrishna, 2020 introduced a deep learning model, specifically a recurrent neural network (rnn). this model utilized document embedding vectors to infer the meaning of small paragraphs comprising one, two, or three sentences. significant challenges arise when representing sentences with low-resource languages (lrls), such as telugu, marathi, and bengali. furthermore, limited datasets make the process of textual similarity detection more challenging in lrls. @xcite focused on two low-resource indian languages (hindi and marathi), and their proposed model was evaluated on real text classification datasets to show embeddings obtained from synthetic data, which will be an effective training strategy for low-resource languages. they achieved the highest score of 0.83 utilizing the mahabert and labse models. a cross-lingual model for finding similarity between sentences was proposed by @xcite their system obtained an accuracy of 0.82 for finding semantic relatedness in low-resource languages like hindi, marathi, kannada, telugu, malayalam, tamil, gujarati, odia, bengali, and punjabi. @xcite proposed a multilingual framework for finding semantic textual similarity in low-resource languages utilizing rich annotation data from a high-resource language. their shared sentence encoder approach archived score of 0.825 for spanish language. ## dataset and task description the dataset is developed by ousidhoum et al. @xcite to evaluate the semantic textual similarity to perform the shard task at semeval-2024. it includes data for telugu, marathi, english, and spanish, which assess semantic relatedness between sentences. there are labeled and unlabeled data in the dataset. labeled data is used for track-a; each row has two sentences and a score that ranges from 0 to 1, representing the semantic relatedness of the sentences. moreover, the dev, test, and train categories are applied to every language dataset. ## system overview various textual features will be extracted to employ ml and dl models. several transformer models are exploited for the task, including minilm-l6-v2, marathi-sbert, telugu-sentence-bert-nli, and roberta-bne-sentiment-analysis-es, to assess semantic relatedness. figure 2 illustrates the schematic structure of the proposed approach. textual feature extraction: feature extraction is necessary for ml and dl models to learn from text. we used tf-idf @xcite to extract the features to apply different ml algorithms. word2vec @xcite and fast-text @xcite embeddings were used to extract features for dl models. ## results and analysis the results demonstrate that the ml models perform poorly. dl models are slightly better, but they need to be better. transformer-based models demonstrated exceptional performance across all languages. for the english language, mpnet-v2, distilbert, and minilm-l12 scored 0.821, 0.821, the marathisbert model performed best, with a spearman correlation score of 0.870. for the telugu language, telugu-sbert and labse had scores of 0.761 and 0.804, respectively. the best model for this language is telugu-bert-nli, which has a spearman correlation rank of 0.804. for spanish, roberta-bne and stsb-xlm have scores of 0.659 and 0.677, respectively, but spanish-bert outperforms both by 0.677. figure 3 illustrates the summary of the best-performed models in four languages of the task. transformer-based outperformed other models by a wide margin for four languages in the task. one explanation is its ability to capture information's bidirectional context. in addition, it produces contextual word representations that make polysemy understandable and enables capturing minute variations in meaning depending on context. since these transformer-based model has been pre-trained on various linguistic tasks using a sizable dataset, fine-tuning these models improves results. ## conclusion this study explores the efficacy of various machine learning (ml), deep learning (dl), and transformer-based models for analyzing semantic relatedness within texts across four languages: english, spanish, telugu, and marathi. experimental assessments reveal subpar performance of both ml and dl models across all languages. however, transformer-based models exhibit superior capabilities in discerning semantic relatedness within the given task. specifically, the minilm-l6 model excels for english, marathisbert for marathi, telugusbert for telugu, and spanishsbert for spanish, achieving peak ρ scores of 0.822, 0.870, 0.820, and 0.677, respectively. the study suggests that augmenting training data could enhance the performance of current models. additionally, leveraging advanced techniques such as large language models (llm) and generative pre-trained transformers (gpt) holds promise for further improvement.