NRK at S em E val-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT -based models (2024)

Abstract: This paper describes the system of the team NRK for Task A in the SemEval-2024 Task 1: Semantic Textual Relatedness (STR). We focus on exploring the performance of ensemble architectures based on the voting technique and different pre-trained transformer-based language models, including the multilingual and monolingual BERTology models. The experimental results show that our system has achieved competitive performance in some languages in Track A: Supervised, where our submissions rank in the Top 3 and Top 4 for Algerian Arabic and Amharic languages. Our source code is released on the GitHub site.

## introduction the semeval-2024 task 1 @xcite aims at detecting the degree of semantic relatedness between pairs of sentences across 14 different languages, encompassing afrikaans, algerian arabic, amharic, english, hausa, hindi, indonesian, kinyarwanda, marathi, moroccan arabic, modern standard arabic, punjabi, spanish, and telugu. this shared task has three main tasks, each focusing on different aspects of predicting semantic textual relatedness within sentence pairs. semantic textual relatedness (str) is a task in natural language processing (nlp) that aims to measure the degree of semantic relatedness between two text passages, typically sentences. str plays a crucial role in various nlp applications, as it allows computers to understand the relationships between different pieces of text. as mentioned in @xcite , it is also employed in chatbots and dialogue systems to understand the user's intent and in question-answering systems to identify answer passages that are semantically related to the question. additionally, str finds applications in text summarization, where it helps identify the most important and semantically relevant sentences to create a concise summary of a longer document. str also plays a role in text generation tasks, such as machine translation and dialogue systems, by guiding the model to generate text that is semantically related to the input or context. however, accurately measuring str presents several challenges. one key challenge lies in capturing the nuances of language, such as synonyms, paraphrases, and ambiguity. another challenge is dealing with different languages and cultural contexts, where semantic relationships might not be directly translatable. our team only focuses on addressing track a in the shared task. our approach is based on the domain adaption for different transformer-based models, and then we continue to fine-tune the pre-trained transformer-based models on the taskspecific training data. therefore, our system is able to leverage domain-specific knowledge to improve performance. subsequently, we train a crossencoder model on the adapted transformer-based models, harnessing its ability to capture semantic relatedness between sentence pairs effectively. to further enhance the robustness and performance of our predictions, we adopt a weighted voting technique to combine the outputs of multiple models. ## experimental setup data and pre-processing: we utilized the official training set for training models. the development set was used to determine the weights for each model chosen to apply the voting technique based on their performance. ## results and discussion in this section, we present the official results of our final submission model for track a in the semeval 2024 task 1, comparing them with the results of the two top-performing teams for each sub-track. ## conclusion this paper introduces a straightforward yet effective ensemble architecture for track a in the semeval-2024 task 1: semantic textual relatedness. our system leverages fine-tuning of pretrained transformer-based language models as base regressors, coupled with a weighted voting technique to amalgamate predictions from diverse base models. experimental results demonstrate its competitive performance across select languages in track a without any additional resources. for future works, we propose enhancing our system by integrating african transformer-based models and exploring data augmentation techniques to improve the overall performance.