W arwick NLP at S em E val-2024 Task 1: Low-Rank Cross-Encoders for Efficient Semantic Textual Relatedness (2024)

Abstract: This work participates in SemEval 2024 Task 1 on Semantic Textural Relatedness (STR) in Track A (supervised regression) in two languages, English and Moroccan Arabic. The task consists of providing a score of how two sentences relate to each other. The system developed in this work leveraged a cross-encoder with a merged fine-tuned Low-Rank Adapter (LoRA). The system was ranked eighth in English with a Spearman coefficient of 0.842, while Moroccan Arabic was ranked seventh with a score of 0.816. Moreover, various experiments were conducted to see the impact of different models and adapters on the performance and accuracy of the system.

## introduction semantic textual relatedness (str) is a measure of how closely related or connected two linguistic units are in terms of their meanings or concepts @xcite . str is a valuable concept in natural language processing (nlp), as it helps us to understand the connections and similarities between different pieces of text. by determining the degree of relatedness between sentences or phrases, we can improve various nlp tasks such as information retrieval, question answering, and text summarisation. this understanding of semantic relatedness enables us to create more accurate word embeddings and sentence representations, enhancing the performance of language processing models. one way to represent str is a supervised regression task in which the output is a continuous score number between 0 and 1. for the str task in semeval 2024 @xcite , the organisers on track a (supervised) provided datasets @xcite for nine languages or dialects. they provided pairs of sentences and annotated the degree of relatedness via a human score between 0 and 1. the languages considered in this work are english and moroccan arabic. these two were selected because they are comprehensive for the team. there are various methods that can be used to estimate the relatedness of two sentences. one of the methods is to utilise the pre-trained language models (plms). plms are currently stateof-the-art in the field of nlp, and follow the transformer architecture introduced in @xcite with the attention mechanism. another variation that uses a mechanism of cross-attention is the cross encoder @xcite , whichthat of takes two inputs and outputs a score between 0 and 1 on how related these two inputs are. they are efficient in determining the correlation between two inputs. parameter-efficient fine-tuning (peft) aims to tune the pre-trained model with high accuracy but with less cost and complexity. one of the peft methods is adapters @xcite , which tune extra parameters or layers instead of tuning the whole model while maintaining competitive accuracy. they can be considered as few-shot learners as per @xcite . one type of adapter is the low-rank adapter (lora). instead of tuning the whole weight, lora adds small matrices in each layer, and these matrices would be fine-tuned. hence, this work applies a tuned lora adapter on a pre-trained cross-encoder to estimate the score of the relatedness of two sentences. the code is publicly available foot_0 . the paper is organised as follows: section 2 presents the background, including related work and dataset overview; section 3 covers the system overview; section 4 presents the results; section 4 discusses the error analysis and limitations; and the paper concludes. ## background this section will cover the related work, dataset description and a brief introduction to peft. ## system development this section covers the cross-encoder, lora adapter, the developed system architecture, and the evaluation metric. ## results the system would initially be trained on the training set and evaluated with the development set, and the official results would be produced by applying the model to the testing sets. this work's approach is incremental and experimental. so, variations of models and adapters would be tested, and the models achieving better metrics would be selected.