NLP _ T eam1@ SSN at S em E val-2024 Task 1: Impact of language models in Sentence- BERT for Semantic Textual Relatedness in Low-resource Languages (2024)

Abstract: Semantic Textual Relatedness (STR) will provide insight into the limitations of existing models and support ongoing work on semantic representations. Track A in Shared Task-1, provides pairs of sentences with semantic relatedness scores for 9 languages out of which 7 are low-resources. These languages are from four different language families. We developed models for 8 languages (except for Amharic) in Track A, using Sentence Transformers (SBERT) architecture, and fine-tuned them with multilingual and monolingual pre-trained language models (PLM). Our models for English (eng), Algerian Arabic (arq), andKinyarwanda (kin) languages were ranked 12, 5, and 8 respectively. Our submissions are ranked 5th among 40 submissions in Track A with an average Spearman correlation score of 0.74. However, we observed that the usage of monolingual PLMs did not guarantee better than multilingual PLMs in Marathi (mar), and Telugu (tel) languages in our case.

## introduction prior nlp work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. the first dataset for semantic textual relatedness, str-2022 was introduced by @xcite english sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. the semantic relatedness of two units of language is the degree to which they are close in terms of their meaning @xcite . the linguistic units can be words, phrases, sentences, etc. the most semantic similarity datasets were annotated using coarse rating labels such as integer values between 1 and 5 representing coarse degrees of closeness. these datasets suffer from issues arising due to the fixed granularity which intuits fuzzy boundaries between related and unrelated notions. the following subsection describes the difference between similarity and relatedness which is crucial in understanding the textual semantics. ## related work similarity task is originally proposed to mimic human perception of the similarity level between word or sentence pairs. the first, word similarity dataset was collected in @xcite , which consisted of 65-word pairs with human annotations. in general, the datasets consist of pairs of words (w1, w2) (or sentences) and human-annotated similarity scores s h . @xcite measured the semantic relatedness using contextual versus static embeddings and unsupervised versus supervised approach to sentence representation. in an unsupervised approach, the embedding of a sentence is derived from that of its constituent tokens. they used word2vec, glove, and fasttext static embeddings in unsupervised settings and the majority of the static embedding models failed to obtain better correlations with human annotation scores. the contextual embeddings from bert and roberta do not perform better than the word2vec embeddings. finally, the supervised approach by finetuning the sbert with the str-2022 dataset captured high semantic relatedness and the spearman correlation is 0.82 and 0.83 for bert-based and roberta-based respectively. the supervised approach using the sbert framework by formulating a regression task leads to a better correlation score of 0.20 than the unsupervised approach. this motivated us to use the sbert framework to score the semantic relatedness between the pairs of sentences across 6 low-resource languages and english, and spanish in the track a dataset. we used 2 multilingual pre-trained language models (labse, pp-mpnet-v2) and language-specific monolingual lm for each of the languages. the following subsections describe the reason behind the selection of particular pre-trained lms that are used in our models. ## system overview given a human-annotated dataset for semantic textual relatedness, the participants are allowed to submit systems that have been trained using the labeled training datasets. apart from that, the participating teams are also allowed to use any other publicly in our experiment, predicting semantic relatedness is treated as a regression task, where each sentence is represented as a vector. we use the cosine similarity between the vectors to predict their semantic relatedness, s p , the spearman score predicted by the system. finally, the correlation between s h , the spearman score manually annotated by humans, and s p is computed, and a higher correlation suggests good alignment with human annotations and a better embedding model. usually, the spearman correlation between the prediction and gold relatedness scores is used to measure the goodness of the relatedness predictions. ## experimental setup we aim to focus on the impact of the sentence-bert deep neural network in semantic textual relatedness scoring tasks, and the benefit of multilingual/monolingual pre-trained lms over the task especially for the low-resource languages. ## result during the testing phase, we combined the training + development data as training data to fine-tune the model that yielded the maximum score during the development phase. then the model is tested with the test dataset of the corresponding language. the predicted sentence relatedness score by the models is submitted as a result and is evaluated using the spearman coefficient. the results are shown in the sbert-based models for the languages english (eng), algerian arabic (arq), and kinyarwanda (kin) performed more than the baseline spearman score. sbert-labse model for spanish (esp) scored almost equal to the baseline system. even though the monolingual models sbert-mahasbert for marathi and sbert-telugusbert for telugu showed better performance during the development phase, failed to score above the baseline during testing in respective languages. similarly, sbertbased models trained using multilingual pre-trained lm for moroccan arabic (ary) and hausa (hau) languages scored lesser than the baseline model in the test phase.