MBZUAI - UNAM at S em E val-2024 Task 1: Sentence- CROBI , a Simple Cross-Bi-Encoder-Based Neural Network Architecture for Semantic Textual Relatedness (2024)

Abstract: The Semantic Textual Relatedness (STR) shared task aims at detecting the degree of semantic relatedness between pairs of sentences on low-resource languages from Afroasiatic, Indoeuropean, Austronesian, Dravidian, and Nigercongo families. We use the Sentence-CROBI architecture to tackle this problem. The model is adapted from its original purpose of paraphrase detection to explore its capacities in a related task with limited resources and in multilingual and monolingual settings. Our approach combines the vector representation of cross-encoders and bi-encoders and possesses high adaptable capacity by combining several pre-trained models. Our system obtained good results on the low-resource languages of the dataset using a multilingual fine-tuning approach.

## introduction task 1 of semeval 2024 @xcite focuses on semantic textual relatedness (str). given two sentences, the semantic relatedness between them is defined as the degree of closeness between their meanings @xcite . however, the traits that make two sentences to be understood as related entities can be of different order, such as the underlying syntactic structure, lexical affinity, or the author's style, among others. the task organisers have chosen for this track a set of languages, among which english and spanish stand out, two languages with numerous computational resources. the rest, are low-resourced languages from africa (algerian arabic, moroccan arabic, amharic, hausa, kinyarwanda) and asia @xcite . three tracks were proposed in the task: supervised, unsupervised and cross-lingual. we participated in track a, supervised. this is a regression problem since a relatedness coefficient must be given that ranges from 0 to 1 for each pair of sen-tences. our solution is based on using the sentence-crobi model, introduced in ortiz-barajas et al. @xcite , which was designed for paraphrase detection with very good results in english. our hypothesis is that the same methods used in paraphrase detection can be applied to the determination of the degree of relatedness. the structure of the paper is the following. in section 2, we describe the related work on this task using pre-trained language models. section 3 briefly describes the dataset. in section 4, we present our methodology. finally, we present our results in the development and evaluation phases in section 5 and conclusions in section 6. ## related work the sentence-bert model @xcite is an approach that generates semantically meaningful sentence embeddings. by training bert on siamese and triplet network structures, this approach is able to capture sentence similarity more effectively. it also reduces computational overhead compared to bert @xcite and roberta @xcite while maintaining high accuracy in tasks such as semantic textual similarity and transfer learning. following this research line, there is an approach to improve bert-based semantic embeddings for similarity tasks @xcite . the authors propose a flow-based calibration method by transforming the original bert embeddings into an isotropic latent space using flow. the proposed method aligns better with gold semantic similarity and reduces the influence of lexical similarity. in this work, we use the sentence-crobi model, a simple architecture that combines bi-encoders and cross-encoders that was originally proposed to solve paraphrase detection. due to its implementation facility, we adapt this model for the semantic relatedness task by only changing the task-specify block, the loss function and the pre-trained models for the cross-encoder and bi-encoder components. ## corpora we briefly describe the corpora that we use to evaluate our model in the semeval shared task 1 in this section. the semrel2024 dataset @xcite is a comprehensive collection of semantic textual relatedness datasets for 14 languages, predominantly spoken in africa and asia. these languages cover a wide range of language families and include both high-resource and low-resource languages. each dataset consists of sentence pairs annotated by native speakers with relatedness scores ranging from 0 (completely unrelated) to 1 (maximally related). the datasets were curated by selecting pairs from various sources such as news data, wikipedia, and conversational data to ensure diversity in topics and formality levels. the relatedness scores were generated through best-worst scaling (bws) annotations, enhancing the reliability of the rankings. it can be noticed it is a highly unbalanced dataset. only english has more than 2,000 training examples, followed by hausa, spanish, algerian arabic, marathi and telugu with more than 1,000 instances and amharic, moroccan arabic and kinyarwanda with less than 1,000 examples. ## methodology in this section, we describe the proposed architecture, the experimental configuration and the training details. for pre-processing the sentence pairs, we perform the same text pre-processing steps as mentioned in (ortiz-barajas et al., 2022). ## results we present the results of our proposed model in the following section in the development and evaluawe report an average of 5 runs in the validation splits used for cross-validation. we obtain the final score predictions in the development set using the bagging technique. ## conclusions this work presents the sentence-crobi model and its adaptation to the semeval 2024 task 1: semantic textual relatedness. we evaluate the model's capacities in monolingual and multilingual fine-tuning approaches to measure its performance and adaptability across diverse linguistic families, yielding acceptable performance in low and mid-resource languages. ensemble techniques further enhance the robustness and reliability of the model's predictions. overall, the findings underscore the model's capacity for solving relatedness detection tasks, emphasising its versatility in accommodating linguistic variations and resource constraints. clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. seid muhie yimam, abinew ali ayele, gopalakrishnan venkatesh, ibrahim gashaw, and chris biemann. 2021. introducing various semantic models for amharic: experimentation and evaluation with multiple tasks and datasets. future internet, 13(11).