AA da M at S em E val-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatedness (2024)

Abstract: This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages. The shared task aims at measuring the semantic textual relatedness between pairs of sentences, with a focus on a range of under-represented languages. In this work, we propose using machine translation for data augmentation to address the low-resource challenge of limited training data. Moreover, we apply task-adaptive pre-training on unlabeled task data to bridge the gap between pre-training and task adaptation. For model training, we investigate both full fine-tuning and adapter-based tuning, and adopt the adapter framework for effective zero-shot cross-lingual transfer. We achieve competitive results in the shared task: our system performs the best among all ranked teams in both subtask A (supervised learning) and subtask C (cross-lingual transfer).

## introduction semantic textual relatedness (str) measures the closeness of meaning between two linguistic units, such as a pair of words or sentences @xcite . for example, one can easily tell that "i like playing games" is more semantically related to "the game is fun" rather than "the weather is good", which largely depends on their lexical semantic relation and topic consistency. semantic textual similarity (sts), a closely related concept, indicates whether two units have a paraphrasing relation. the difference between these two concepts is clarified in @xcite : while similar pairs are also related, the reverse is not necessarily true. in stark contrast to the extensive research on sts @xcite @xcite , exploration of str lags behind and predominantly focuses on 1 our code: https://github.com/uds-lsv/aadam english @xcite , mainly due to the lack of datasets. to close this gap, the semeval-2024 task 1: semantic textual relatedness @xcite is proposed to encourage str research on 14 african and asian languages. the shared task consists of 3 subtasks: supervised (subtask a), unsupervised (subtask b), and cross-lingual (subtask c). in this paper, we present our system aadam (augmentation and adaptation for multilingual str) developed for subtask a and c. our system adopts a cross-encoder architecture which takes the concatenation of a pair of sentences as input and predicts the relatedness score through a regression head @xcite . as the provided task data for non-english languages is relatively limited, we perform data augmentation for these languages via machine translation. to better adapt a pre-trained model to the str task, we apply task-adaptive pre-training @xcite which has shown effectiveness on many tasks @xcite . for subtask a, we explore full fine-tuning and adapterbased tuning @xcite combined with previously mentioned techniques. additionally, we use the adapter framework mad-x @xcite for cross-lingual transfer in subtask c. we select the best model based on the performance on development sets for the final submission, and our system achieves competitive results on both subtasks. in subtask a, our system ranks first out of 40 teams on average, and performs the best in spanish. in subtask c, our system ranks first among 18 teams on average, and achieves the best performance in indonesian and punjabi. ## semrel dataset to encourage str research in the multilingual context, @xcite introduce sem-rel, a new str dataset annotated by native speakers, covering 14 languages from 5 distinct language families. these languages are mostly spoken in africa and asia, and many of them are under-represented in natural language processing resources. as shown in figure 1 , the data sizes vary widely from language to language constrained by the availability of resources. notably, english data comprises 32% of the whole dataset and surpasses other languages by a large margin. ## system overview our system employs a cross-encoder architecture, which takes the concatenation of a pair of sentences as input and predicts the relatedness score through a regression head. compared to bi-encoders @xcite , which extract individual sentence representations and then compare them using cosine similarity, crossencoders generally perform better, at the cost of increased inference latency @xcite . we select cross-encoder because of its superior performance (see appendix a), and leave the exploration of an efficient alternative as future work. the core techniques underlying our system are (i) data augmentation using machine translation ( §3.1), and (ii) task-adaptive pre-training on unlabeled task data ( §3.2). we explore two training paradigms for supervised learning combined with the aforementioned techniques, i.e., fine-tuning and adapter-based tuning ( §3.3), and the latter is also employed for cross-lingual transfer ( §3.4). ## conclusion in this paper, we introduce our multilingual str system, aadam, developed for the semeval-2024 task 1, which achieves competitive results in both subtask a and subtask c. we see noticeable improvements by using data augmentation and taskadaptive pre-training, and demonstrate that adapterbased tuning is an effective approach for supervised learning and cross-lingual transfer. despite these strengths, our fine-grained analysis reveals that capturing nuanced semantic relationships remains a challenge, highlighting the need for further granular investigation and modeling improvements.