CAILMD -23 at S em E val-2024 Task 1: Multilingual Evaluation of Semantic Textual Relatedness (2024)

Abstract: The explosive growth of online content demands robust Natural Language Processing (NLP) techniques that can capture nuanced meanings and cultural context across diverse languages. Semantic Textual Relatedness (STR) goes beyond superficial word overlap, considering linguistic elements and non-linguistic factors like topic, sentiment, and perspective. Despite its pivotal role, prior NLP research has predominantly focused on English, limiting its applicability across languages. Addressing this gap, our paper dives into capturing deeper connections between sentences beyond simple word overlap. Going beyond English-centric NLP research, we explore STR in Marathi, Hindi, Spanish, and English, unlocking the potential for information retrieval, machine translation, and more. Leveraging the SemEval-2024 shared task, we explore various language models across three learning paradigms: supervised, unsupervised, and cross-lingual. Our comprehensive methodology gains promising results, demonstrating the effectiveness of our approach. This work aims to not only showcase our achievements but also inspire further research in multilingual STR, particularly for low-resourced languages.

## introduction the ever-increasing diversity of online content demands robust natural language processing (nlp) techniques that can grasp the nuances of meaning across diverse languages. semantic textual relatedness (str) plays a crucial role in achieving this goal by delving beyond superficial lexical similarity and capturing the deeper connections between sentences. unlike semantic similarity, which focuses solely on the taxonomic overlap of words, str encompasses both linguistic elements and nonlinguistic factors like the topic, point of view, and period. this richer understanding unlocks significant potential in various nlp tasks, regardless of the user's native language. imagine searching for information online in your native language and receiving results that truly understand your intent, and not just match keywords. str holds the key to unlocking this dream, bridging the language gap, and fostering true multilingual communication. despite the recognized importance of semantic textual relatedness (str) for multilingual communication, most prior nlp research has focused on semantic similarity within english due to limitations in labeled data for diverse languages @xcite . this narrow focus restricts the potential of str applications like information retrieval across languages with different cultural contexts or machine translation that accurately captures nuances beyond direct word equivalents. existing relatedness methods primarily target english @xcite , with limited exploration in languages like german, chinese, and japanese @xcite @xcite . this highlights a critical gap in natural language processing (nlp): accurately measuring semantic relatedness across diverse languages. the identified gap in multilingual str research, with its limitations in diverse language applications, demands innovative solutions. this paper dives into the exciting realm of bridging this gap through multilingual semantic textual relatedness (str). specifically, we explore methods to capture the semantic connections between texts in languages like english, marathi, hindi, and spanish. our research focuses on the semeval-2024 shared task, which provides three tracks to evaluate str techniques: for each track, we present a comprehensive methodology, employing diverse language models and rigorously analyzing their performance. this allows us to identify the most effective approaches for each challenge. notably, our submissions achieved promising scores on several tracks, demonstrating the strength and potential of our proposed methods. looking beyond our achievements, this work aims to inspire further exploration of multilingual str, particularly for under-resourced languages. we believe that larger datasets and broader language coverage hold immense potential to benefit the nlp community, unlocking the true potential of language understanding and empowering communication across diverse cultures. ## related work semantic textual relations (str) play an important role in natural language processing (nlp), which aims to identify the degree of semantic similarity between text groups. it forms the backbone of various nlp tasks such as information retrieval, question answering, and paraphrase detection, necessitating the assessment of similarity between sentences, phrases, or documents. historically, detailed str research from the 1900s through the 2000s relied heavily on statistical methods heavily dependent on lexical databases like wordnet. however, these methods suffered from a lack of real-world knowledge integration @xcite . classified translation emerged with developments such as glove @xcite , word2vec, and fasttext, which enabled text to be converted into word input. the current methods require converting corpora into words or sentence-embedded forms and computing connectivity scores. notably, large language models (llms) such as sentence-bert often use cosine similarity in embedded sentences to measure relatedness @xcite . previous methodologies have delved into both knowledge-based (ontology, classification) and corpus-based (unsupervised learning) approaches. for example, @xcite examined three approaches: semantic linkage, classification similarity, and hybrid approaches. notably, the multilingual approach of @xcite , improved by 47%, confirming the potential of emphasis on the use of multilingual strategies. furthermore, studies on less resourceful african languages highlight the need for different data types and methodologies @xcite . a significant challenge in str lies in the scarcity of huge-scale, promising datasets for education and assessment. initiatives like semeval play a pivotal role in addressing this gap through dedicated shared tasks focused on str @xcite . these collaborative efforts foster the improvement and evaluation of str models throughout diverse linguistic landscapes and domain names. the current advent of the str-2022 dataset by @xcite marks a significant leap forward in str studies. this annotated dataset, comprising sentence pairs with relatedness scores, serves as a precious aid for schooling and evaluating str fashions. covering various domains and languages, it displays the multilingual nature of str studies @xcite . moreover, str-2022 addresses biases and perceptions in relatedness judgments. through meticulous curation, it aims to mitigate biases and ensure annotation quality, thereby fostering fair evaluations and robust model development. additionally, the dataset highlights the relative nature of relatedness ratings, emphasizing the significance of context and assignment-precise thresholds in decoding similarity measures @xcite . ## system description in this section, we aim to outline our system's components for assessing semantic textual relatedness across different datasets: a) labeled datasets using supervised learning, b) unlabeled datasets employing unsupervised learning, and c) cross-lingual datasets. we'll detail the utilized data, the models employed in each track, and the results obtained from training these models on respective datasets. ## experimental setup the dimensions of the dataset splits are summarized in ## results tables 2, 3, and 4 present the performance results for the three setups (supervised, unsupervised, and cross-lingual) in our experiments. each table summarizes the f1 score, accuracy, and recall achieved by various models for each language. this table shows the performance of models in the supervised learning setup, where labeled data was available for training. bert-based models ("bert-nli" and "marathi-nli") consistently outperform other models (svr, xgboost) in both english and marathi, achieving significantly higher correlation scores (0.823 and 0.871, respectively). interestingly, the marathi-specific nli model even surpasses the multilingual bert performance in marathi, suggesting the benefit of languagespecific models. this table presents the results for the unsupervised learning setup, where models were trained without relying on labeled data. in english, the bert-based model ("bert-base-uncased" foot_3 ) outperforms the pre-trained sentence-t5 foot_4 model, possibly due to its larger size and fine-tuning on relevant nlp tasks. in hindi, while both indic-bert @xcite and hindi-bert-v2 @xcite have similar f1 scores (around 66%), the latter achieves a significantly higher correlation coefficient (0.796). this indicates that hindi-bert-v2 foot_5 captures semantic relatedness more effectively despite similar overall accuracy. this table shows the performance of models in the cross-lingual learning setup, where the goal was to assess semantic relatedness across different languages. both models used ("all-mpnet-base-v2" for spanish-to-english and "hindi-sentence-bert-nli" for english-to-hindi) achieve worthy correlation scores (0.786 and 0.809, respectively) demonstrating the potential of cross-lingual approaches. ## conclusion in this paper, we presented a comparative analysis of systems for the semantic textual relatedness (str) task at semeval-2024 task 1. our approaches, primarily based on language-specific transformer models, achieved top scores on several tracks, including 1st place in unsupervised learning for hindi. notably, we did not utilize any external datasets, highlighting the effectiveness of our approach despite potential variations in pre-trained model training data. prior research in str has largely focused on english due to limited labeled data for diverse languages. this restricts the true prospect of str applications like multilingual information retrieval and machine translation. we addressed this gap by exploring solutions for str in english, marathi, hindi, and spanish. we aim to inspire further research on multilingual str, particularly for lowresourced languages. we believe larger datasets and broader language coverage hold immense potential for multilingual nlp, unlocking a deeper understanding and empowering cross-cultural communication.