H ausa NLP at S em E val-2024 Task 1: Textual Relatedness Analysis for Semantic Representation of Sentences (2024)

Abstract: Semantic Text Relatedness (STR), a measure of meaning similarity between text elements, has become a key focus in the field of Natural Language Processing (NLP). We describe SemEval-2024 task 1 on Semantic Textual Relatedness featuring three tracks: supervised learning, unsupervised learning and cross-lingual learning across African and Asian languages including Afrikaans, Algerian Arabic, Amharic, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. Our goal is to analyse the semantic representation of sentences textual relatedness trained on mBert, all-MiniLM-L6-v2 and Bert-Based-uncased. The effectiveness of these models is evaluated using the Spearman Correlation metric, which assesses the strength of the relationship between paired data. The finding reveals the viability of transformer models in multilingual STR tasks.

## introduction the rapid increase in digital information has presented a critical challenge for researchers. the web hosts around 50 million pages of text, which is beyond the capacity of human interpretation alone. to interpret this extensive text data effectively, it is essential to comprehend the meanings of various words @xcite . semantic text relatedness (str) is a semantic analysis of the relationship between two pieces of text based on their meanings. str of two language units has long been considered fundamental to understanding meaning @xcite ), it's a metric used to measure the similarity in meaning between two terms or documents. it is a subset of computational linguistics and one of the fundamental concepts of natural language processing (nlp). str can be measured using datasets designed by experts, which are made up of word pairs that are known to be related. it can be used in identifying a paraphrase or duplicate, as well as search engines to give users relevant and personalized results. when two sentences have a paraphrase or entailment relation, they are considered to be semantically similar and when evaluating the semantic relatedness between them, humans typically focus on identifying shared meanings. in the case of the sentence pairs below, most english speakers would agree that the sentences in the first pair are more closely related in meaning than those in the second pair, whether they are from the same topic, express the same view or originate from the same time period etc. @xcite . pair 1: a. there was a lemon tree next to the house. b. the boy enjoyed reading under the lemon tree. pair 2: a. there was a lemon tree next to the house. b. the boy was an excellent football player. previous nlp research has mainly dealt with semantic relatedness primarily in english language. however, in this task, we address a variety of languages, including afrikaans, algerian arabic, amharic, hausa, hindi, indonesian, kinyarwanda, marathi, moroccan arabic, modern standard arabic, punjabi, spanish, and telugu. the task featured the following tracks: track a which is a supervised learning, track b is an unsupervised learning and track c is a cross-lingual learning. ## related works sentences are considered semantically related when they share commonalities in meaning, such as paraphrasal or entailment relations. a study by @xcite developed a semantic textual relatedness dataset (str-2022) to manually annotate english sentence pairs and explore the factors that contribute to the semantic relatedness of sentences. the dataset has been used to study the degree of semantic relatedness and the reliability of human intuition in determining the relatedness of sentence pairs while @xcite assessed the methods for semantic relatedness between words based on knowledge sources. these methods exploit features from both structural and statistical approaches, emphasizing on semantic representation, measures of semantic similarity, and knowledge-based text mining. (lastra-díaz and garcía-serrano, 2015) proposed explicit semantic analysis (esa), a recently introduced approach that signifies the meaning of texts by computing the semantic relevance of natural language texts. this approach assumes the need for substantial amounts of common sense and domain-specific knowledge, utilizing machine learning techniques to explicitly depict the meaning of any text. this is achieved by creating a weighted vector based on concepts from wikipedia. esa undergoes continuous development, ensuring a consistent expansion of its breadth and depth over time. ## experiment and evaluation this section describes the system overview which comprises the dataset description, model description and evaluation metric. ## results and discussion this section presents the results of the shared task tracks. in supervised learning track a, the multilingual bert (mbert) model was used. the model demonstrated different levels of performance across the languages. notably, mbert exhibited strong correlation scores in languages such as english with 0.76, marathi with 0.81, and telugu with 0.79 correlation. this indicates the model's ability to generalize well across linguistic contexts in semantic relatedness tasks. these findings suggest that mbert can effectively capture semantic relatedness, even in low-resource languages, highlighting its robustness and cross-lingual generalization capabilities. however, challenges were observed in languages with complex morphological structures, underscoring the need for further research to address such linguistic nuances. conversely, the unsupervised learning track b featured the all-minilm-l6-v2 model, which achieved promising results in certain languages, particularly english with 0.82, spanish with 0.60, and moroccan arabic with 0.5 spearman correlation value. despite its effectiveness, the model faced difficulties in languages such as punjabi and amharic, where semantic relatedness was harder to capture without labelled data. these challenges emphasize the importance of developing techniques to improve unsupervised learning models' performance, especially in low-resource language settings. similarly, track c (cross-lingual) which were entirely trained with bert-based-uncased performed promisingly despite training and predicting on different language pairs. the spearman correlation for spanish achieved 0.81 and was trained on english, while hausa achieved the lowest with 0.63 despite being trained on kinyarwanda training dataset. this performance especially in semantic textual relationship shows that cross-lingual hold a prospective future for generalization of nlp tasks. however, the findings highlight the effectiveness of transformer models, in capturing semantic relatedness across diverse languages. the choice of evaluation metrics, such as spearman correlation, proved instrumental in assessing the models' performance and understanding their ability to capture the ordinal relationship between predicted and true semantic relatedness scores. furthermore, the results contribute valuable insights into advancing the understanding and application of semantic textual relatedness in multilingual nlp tasks, paving the way for future research in this domain. ## conclusion and future works the study on semantic text relatedness (str) across multiple languages has demonstrated the effectiveness of transformer models in capturing semantic relatedness. the multilingual bert (mbert) model showed strong correlation scores in languages such as english, marathi, and telugu, indicating its ability to generalize well across linguistic contexts. the all-minilm-l6-v2 model achieved promising results in english, spanish, and moroccan arabic, while facing challenges in languages like punjabi and amharic. the crosslingual track, using bert-based-uncased, also performed well, especially in spanish, trained on english data. these findings underscore the potential of transformer models in nlp tasks and the importance of appropriate evaluation metrics like spearman correlation. the study contributes valuable insights into advancing semantic textual relatedness in multilingual nlp, highlighting areas for future research and development. future work should focus on exploring advanced transformer large language models (llms) like gpt-3 and t5 to improve performance across diverse languages, including low-resource ones. expanding language coverage, incorporating contextual and cultural information, and fine-tuning with language-specific data will enhance model accuracy. cross-lingual transfer learning techniques can be investigated to adapt high-resource language models to low-resource settings. hybrid approaches combining different learning methods may offer improved results, while new evaluation metrics could better capture semantic nuances. additionally, exploring multimodal str and applying research findings to real-world applications will increase the practical impact of str systems.