YSP at S em E val-2024 Task 1: Enhancing Sentence Relatedness Assessment using S iamese Networks (2024)

Abstract: In this paper we present the system for Track A in the SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages (STR). The proposed system integrates a Siamese Network architecture with pre-trained language models, including BERT, RoBERTa, and the Universal Sentence Encoder (USE). Through rigorous experimentation and analysis, we evaluate the performance of these models across multiple languages. Our findings reveal that the Universal Sentence Encoder excels in capturing semantic similarities, outperforming BERT and RoBERTa in most scenarios. Particularly notable is the USE’s exceptional performance in English and Marathi. These results emphasize the importance of selecting appropriate pre-trained models based on linguistic considerations and task requirements.

## introduction semantic relatedness is a fundamental measure in natural language processing, providing a detailed assessment of how closely related two pieces of text are on a semantic level. this metric has wideranging significance in nlp tasks such as information retrieval, question answering, and text summarizing, contributing to the understanding of textual connections and improving algorithmic performance. the importance of determining meaning through quantifying relatedness has been acknowledged within linguistic discussions for many years. automating the determination of connected meanings holds significant value across diverse applications like evaluating sentence representation methods or supporting question-answering systems as well as summarizing processes. however, the application of measuring relatedness to non-english languages presents substantial challenges due to disparities in linguistic resources and annotated datasets. in contrast to english many languages lack comprehensive lexical or syntactic resources which creates challenges in inaccurately capturing semantic subtleties and establishing cross-language associations. additionally, variations across different languages such as morphological, syntactic, and semantic also add complexity to developing language-independent models for expressing relatedness overcoming these obstacles requires extensive research and effective resource development tailored to different linguistic contexts ensuring that measures offered by the semantic similarity can be effectively applied across different languages. here are some examples of the score of semantic relatedness of two sentences in three languages: english: "you figure this out all by yourself, did you? did you find all this on your own?" score: 0.88 spanish: "jean hebb swank es una astrofísica conocida por sus estudios sobre agujeros negros y estrellas de neutrones. bajo la supervisión de steve frautschi, obtuvo su doctorado en física en 1967." score: 0.52 kinyarwanda: "east africa's got talent ku nshuro yayo ya mbere u rwanda ni kimwe mu bihugu byemerewe kuyitabira aho rwahuriye n'ibindi birimo uganda, tanzania na kenya. iya mbere ni umubano mwiza uri hagati ya mali n'u rwanda, ndetse u rwanda ni kimwe mu bihugu bifite abapolisi bagiye kugarura amahoro muri mali." score: 0.09 the rest of this paper is structured as follows: section 2 introduces the problem statement and provides a summary of related works. sections 3 and 4 detail the system description and experimental setup, respectively. the evaluation results are outlined in section 5, followed by our conclusion ## system overview the measurement of semantic relatedness involves assessing the connection or correlation between words or phrases, regardless of their meanings. recently, deep learning models like convolutional neural networks and recurrent neural networks have garnered attention in measuring semantic relatedness. these models excel at capturing intricate patterns and interdependencies in textual data, thereby enhancing performance across a range of natural language processing tasks. for instance, siamese networks make use of cnn or rnn structures to compare embeddings and gauge the relatedness between sentences or short texts. attention mechanisms have also been integrated into these models to improve focus on crucial semantic elements @xcite . siamese architectures are effective because they use the same model to handle similar inputs, and makes it easier to compare sentence pairs and reduces the number of parameters that need training, requiring less data and making them less susceptible to overfitting @xcite . we implement a comprehensive system for evaluating sentence relatedness using a siamese network architecture coupled with universal sentence encoder (use) embeddings @xcite . the system comprises several distinct components, each contributing to the overall functionality and effectiveness of the model. the universal sentence encoder model is loaded from tensorflow hub foot_0 , enabling the generation of high-quality embeddings for the input sentences. these embeddings capture semantic information, thereby facilitating the comparison and analysis of sentence similarity. both the training and validation sentences are encoded into use embeddings, which are then converted to numpy arrays for seamless integration with the siamese network architecture. the core of the system lies in the construction and training of the siamese network model (figure 1 ). utilizing tensorflow's functional api, the model is designed to accept two input embeddings corresponding to pairs of sentences. it computes the euclidean distance between these embeddings and passes the result through a dense layer with sigmoid activation to predict the similarity score between the sentences. by employing mean squared error (mse) loss and the adam optimizer, the model is trained on the training data, with performance monitored using the validation set. ## results our rankings show that in certain languages, our method closely matches the baseline performance. for example, in english, our rank is 0.82 compared to the baseline of 0.83, and in spanish, our rank stands at 0.64 compared to the baseline of 0.7. therefore, our method demonstrates significant effectiveness across different languages. the evaluation of models across different languages showed varied performances. bert demonstrated strong performance in english and spanish, while roberta excelled in marathi and spanish. however, the universal sentence encoder consistently delivered exceptional results across most languages in the training set ( specifically, the use model achieved the highest scores in english with 0.75 on the training set and 0.82 on the test set, indicating remarkable performance. the top overall score reached 0.86, showcasing its effectiveness in this task as well. following english, marathi exhibited the second-best performance at a score of 0.69 ( ## conclusion the system presented offers a strong framework for evaluating sentence similarity by integrating a siamese network architecture with universal sentence encoder embeddings. a comprehensive overview of the system's components and processes, including data preprocessing, model construction, and training, demonstrates that the system effectively utilizes advanced techniques in natural language processing to make accurate similarity predictions. the evaluation results reveal the superior performance of the universal sentence encoder across multiple languages, outperforming pre-trained models like bert and roberta in most scenarios. notably, the system excelled in english and marathi, demonstrating its versatility and effectiveness across diverse linguistic contexts. further optimization and refinement of the system may enhance its performance in under-performing languages as well as broaden its applicability in real-world scenarios, ultimately advancing the field of natural language processing and facilitating a wide range of practical applications.