TECHSSN at S em E val-2024 Task 1: Multilingual Analysis for Semantic Textual Relatedness using Boosted Transformer Models (2024)

Abstract: This paper presents our approach to SemEval- 2024 Task 1: Semantic Textual Relatedness (STR). Out of the 14 languages provided, we specifically focused on English and Telugu. Our proposal employs advanced natural language processing techniques and leverages the Sentence Transformers library for sentence embeddings. For English, a Gradient Boosting Regressor trained on DistilBERT embeddingsachieves competitive results, while for Telugu, a multilingual model coupled with hyperparameter tuning yields enhanced performance. The paper discusses the significance of semantic relatedness in various languages, highlighting the challenges and nuances encountered. Our findings contribute to the understanding of semantic textual relatedness across diverse linguistic landscapes, providing valuable insights for future research in multilingual natural language processing.

## introduction semantic textual relatedness (str) is a pivotal aspect of natural language processing (nlp) that underlies the foundation of various language-related tasks @xcite . this task takes this challenge to a global scale by encompassing 14 languages, including afrikaans, algerian arabic, amharic, english, hausa, hindi, indonesian, kinyarwanda, marathi, morrocan arabic, modern standard arabic, punjabi, spanish, and telugu. this multilingual approach transcends linguistic boundaries, fostering collaboration and research within the nlp community. exploring a diverse array of languages in this endeavor encourages the development of models adept at handling the distinct linguistic nuances inherent in each language. this progress fosters a more inclusive and universally applicable approach to natural language processing research. semeval-2024 task 1 delves into the automated detection of semantic relatedness between pairs of sentences, a foundational aspect for unraveling meaning. the task's embrace of multiple languages is crucial, encompassing a spectrum of linguistic characteristics. this inclusivity fosters a collaborative atmosphere for researchers, pushing them to craft models adept at capturing semantic nuances across diverse linguistic landscapes. the task's importance extends to the assessment and benchmarking of sentence representation methods, pivotal for numerous nlp applications. str evaluated through tasks, play a crucial role in areas such as question answering, summarization, and information retrieval. the task's outcomes serve as a benchmark, guiding the development and refinement of models that can effectively discern the relatedness of sentences, regardless of language. when approaching this task, we concentrated on two languages: english and telugu. employing the sentence transformers library, we make use of the distilbert model for generating sentence embeddings in english and opt for a multilingual model for handling telugu. the choice of state-of-the-art models and embedding techniques underscores our commitment to developing robust solutions capable of handling the linguistic diversity presented in this task. participating in this task has revealed crucial insights into the intricacies of semantic relatedness across languages. our system demonstrated better performance, particularly in english, achieving noteworthy spearman correlation coefficients on the test set. however, the challenges surfaced in capturing subtle nuances in semantic relations, especially in the context of telugu. this highlights the necessity for specialized methodologies to address the complexities of multilingual semantic relatedness. as we explore the methodology, experiments, and results in the following sections, we delve deeper into the intricacies of our approach, providing a comprehensive understanding of how our model navigates the challenges posed by the task. ## background the task at hand revolves around semantic textual relatedness (str), focusing on evaluating the degree of semantic closeness between sentences in both english and telugu. unlike emotion recognition, this task delves into understanding the relationships between sentences rather than categorizing emotions in code-mixed interactions. in this scenario, the input comprises pairs of sentences in either english or telugu, and the objective is to determine the relatedness score for each pair on a scale from 0 to 1. for instance, consider the following english sentence pair: sentence 1: "the sun is setting over the horizon, casting a warm glow on the city." sentence 2: "as the day comes to an end, the sun sets, and the city is bathed in a warm glow." score label: 0.85 in this example, the relatedness score of 0.85 indicates a high degree of semantic closeness between the two sentences, as they convey similar information about the sunset and the warm glow of the city. sentence 1: "the scientific method involves systematic observation and experimentation." sentence 2: "bicycles are a popular mode of transportation in urban areas." score label: 0.15 in this example, the relatedness score of 0.15 indicates a low degree of semantic closeness between the two sentences. similarly, a dataset exists for telugu. the datasets used for this task include pairs of sentences in english and telugu, capturing the real-world scenario of diverse linguistic interactions. these datasets are annotated with relatedness scores, providing a basis for training and evaluating models effectively. the input parameters consist of the sentence pairs, and the output involves predicting the relatedness score for each pair. the multilingual nature of the task fosters collaboration and research across linguistic boundaries, contributing to a more inclusive and globally applicable approach in the nlp community. ## related work palakorn achananuparp et. al.presents an evaluation of fourteen existing text similarity measures @xcite . the ability to ac-curately judge the similarity between natural language sentences is crucial for various applications such as text mining, question answering, and text summarization. the evaluation encompasses three different datasets: trec9 question variants, microsoft research paraphrase corpus, and the third recognizing textual entailment dataset. the study explores three classes of measures: word overlap, tf-idf, and linguistic measures. the goal is to judge sentence pairs based on the notion that they have identical meanings, considering factors such as paraphrase or entailment. they address the challenges of computing sentence similarity, highlighting the importance of recognizing semantic equivalence beyond surface form comparisons. pantulkar sravanthi and b. srinivasu, address the challenge of measuring sentence similarity, emphasizing the importance of semantic similarity over syntactic measures @xcite . they introduce three semantic similarity approaches-cosine similarity, path-based (wu-palmer and shortest path), and feature-based. the feature-based approach incorporates wordnet, tagging, and lemmatization, showing superior performance in generating semantic scores. this study contributes valuable insights into semantic similarity measures and can enhance the understanding of feature-based approaches based on wordnet in sentence categorization. syed s. akhtar et al. @xcite address the need for word similarity datasets in indian languages, specifically urdu, telugu, marathi, punjabi, tamil, and gujarati. they introduce manually annotated monolingual word similarity datasets for these languages, created through translation and re-annotation of english datasets. the paper presents baseline scores for word representation models using state-of-the-art techniques for urdu, telugu, and marathi, evaluated on the newly created datasets. this work contributes valuable resources for evaluating word representations in indian languages, fostering the development of techniques leveraging word similarity. ## results the system's performance was assessed using a regression model that predicts similarity scores between english sentences. the spearman correlation coefficient is the major quantitative finding, measuring the monotonic relationship between expected and actual similarity scores. the spearman correlation coefficient, often denoted as (rho), is a statistical measure used to assess the strength and direction of the monotonic relationship between two variables. specifically, in the context of evaluating models, the spearman coefficient is employed to quantify the association between predicted scores and actual scores. ## conclusion our system, anchored in the robust capabilities of sentence transformers and gradient boosting regressor models, showcased a better performance in predicting semantic textual relatedness. the fusion of advanced tokenization, embeddings, and hyperparameter tuning resulted in a model finely attuned to the intricacies of the english and telugu languages. the results on the evaluation metrics, particularly the spearman correlation coefficient, underscore the efficacy of our approach in capturing the nuances of semantic relatedness across diverse sentence pairs. the successful adaptation to multiple languages, as evident in the telugu experiments, showcases the versatility of our system. for future work, delving deeper into languagespecific processing techniques and exploring more sophisticated models may unlock additional performance gains. additionally, expanding the system's applicability to handle a broader array of languages and domains could further enhance its utility in real-world applications. overall, our endeavors open avenues for continuous refinement and exploration in the realm of semantic textual relatedness prediction.