NLU - STR at S em E val-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness (2024)

Abstract: Semantic textual relatedness is a broader concept of semantic similarity. It measures the extent to which two chunks of text convey similar meaning or topics, or share related concepts or contexts. This notion of relatedness can be applied in various applications, such as document clustering and summarizing. SemRel-2024, a shared task in SemEval-2024, aims at reducing the gap in the semantic relatedness task by providing datasets for fourteen languages and dialects including Arabic. This paper reports on our participation in Track A (Algerian and Moroccan dialects) and Track B (Modern Standard Arabic). A BERT-based model is augmented and fine-tuned for regression scoring in supervised track (A), while BERT-based cosine similarity is employed for unsupervised track (B). Our system ranked 1st in SemRel-2024 for MSA with a Spearman correlation score of 0.49. We ranked 5th for Moroccan and 12th for Algerian with scores of 0.83 and 0.53, respectively.

## introduction the literature commonly examines semantic similarity, where the focus is on whether two linguistic units (words, phrases, sentences, etc.) share similar meanings @xcite . however, semantic textual relatedness (str) is less explored due to its complexity and the scarcity of datasets @xcite . while the former task checks for the presence of similar meaning or paraphrase, str takes a more comprehensive approach, evaluating relatedness across multiple dimensions, spanning topical similarity, conceptual overlap, contextual coherence, pragmatic connection, themes, scopes, ideas, stylistic conditions, ontological relations, entailment, temporal relation, as well as semantic similarity itself @xcite @xcite . for example, consider the two sentences (the earth orbits the sun at a speed of ~110,000 km/h.) and (earth rotates at ~1670 km/h around its axis.). they hold semantic relatedness through the shared topic of earth's speeds. in contrast, both sentences are not semantically similar as they possess distinct meanings. this illustrates the broader range of str as described by @xcite , which ranges from highly relevant sentences, expressing the same idea with different wording, to entirely unrelated sentences, discussing unrelated topics. semantic relatedness has proven to be useful in evaluating sentence representations generated by language models @xcite , in addition to question answering @xcite , machine translation @xcite , plagiarism detection @xcite , word-sense disambiguation @xcite , among others. exploring the relatedness and similar tasks in languages other than english is hindered by the lack of data @xcite . the semrel-2024 shared task @xcite provided datasets in fourteen languages and offered three tracks. in the supervised track (a), training and testing are performed on the same language. in the unsupervised track (b), the use of labeled data for training is prohibited; and in the cross-lingual track (c), testing is conducted on a different language than the one used for training. this paper presents our contribution to track a and track b. in track a, we fine-tuned bert models using the algerian and moroccan sentence pairs to produce similarity scores. to enrich the data, we augmented the semrel-2024 dataset @xcite by generating additional sentence pairs from google gemini 1 , a generative model, using a predefined prompt template. these generated pairs imitated the style and meaning of the existing pairs, and we assigned them scores corresponding to the originals. we used the same datasets provided by the shared task in addition to a ~760 augmented moroccan pairs to finetune bert models, arabertv2 @xcite and arbertv2 @xcite , which resulted in a performance enhancement of 0.05 points. in track b, as training on labeled data is not allowed, we used cosine similarity using average pooling embedding @xcite on top of each model. our approaches achieved spearman scores @xcite of 0.49 for msa (ranked first), 0.83 for moroccan (ranked fifth), and 0.53 for algerian (ranked twelfth). ## related work semantic textual relatedness (str) has proven to be a valuable task in numerous nlp applications, including the evaluation of llms @xcite . determining the degree of relatedness in str, however, remains a challenging task in computational semantics. that is because str encompasses a broader range of commonalities beyond just meaning, including shared viewpoint, topic, and period, demanding a deeper understanding than semantic similarity alone @xcite . for example, consider reading these two sentences (he heard the waves crashing gently) and (making him feel calm and peaceful). while humans easily recognize their strong relatedness and shared description of the same view (a beach scene), machines require advanced lexical and statistical methods to achieve the same level of understanding. str techniques mainly come from four approaches: lexical similarity @xcite @xcite , semantic similarity @xcite , deep learning @xcite , and llms @xcite . recently, @xcite introduced their str-2022 dataset, which uses fine-grained scores ranging from 0 (least related) to 1 (completely related). their dataset consists of 5,500 scored english sentence pairs. they framed the task as supervised regression, where they fine-tuned two language models, bert-base (kenton and toutanova, 2019) and roberta-base @xcite , and applied average pooling on top of the final embedding layer. their testing of these models on the str-2022 dataset yielded an average spearman correlation of 0.82 for bert-base and 0.83 for roberta-base. on the other hand, their unsupervised experiments using word2vec @xcite achieved a correlation score of 0.60, outperforming both bert-base (0.58) and robertabase (0.48) by 0.02 and 0.12 points, respectively. asaadi et al. (2019 ) created the bi-gram semantic relatedness dataset (bird) for examining semantic composition. to avoid inconsistencies and biases from traditional 1-5 rating scales, they employed fine-grained scoring of bi-gram pairs (0-1) using the best-worst scaling (bws) annotation technique @xcite . the dataset consists of 3,345 scored english term pairs. they utilised three models to generate word representations: glove @xcite , fasttext @xcite , and a word-context co-occurrence matrix @xcite . to calculate relatedness scores between pairs, they employed cosine similarity between the generated addition-pooled vectors. the fasttext model achieved the highest performance with a pearson correlation of 0.60. the semantic relatedness between noun-pairs was studied using contextual similarity by @xcite . they attempted to understand distinctions between nouns in contextual discourse and how the similarity can be broader than just the meaning. additional ideas could rely on extracting named entities @xcite to measure the relatedness @xcite . however, the task evolved, leading to the creation of the up-to-date dataset presented by the semrel-2024 shared task @xcite . their dataset annotation scores are at the level of sentence pairs. they shared baseline results for fourteen languages and dialects using spearman correlation score. since our focus is on arabic, we have chosen its results to show. for example, their baseline is 0.42 for msa in track b using multilingual bert (mbert) (kenton and toutanova, 2019), 0.60 for algerian and 0.77 for moroccan in track a using label agnostic bert sentence embeddings (labse) @xcite . specifically, their algerian arabic dataset offers 1,261 training and 583 test instances, moroccan arabic dataset includes 924 training and 425 test instances, and msa arabic dataset has 595 instances for testing. many efforts have been made to understand arabic dialects, such as dialect identification, intent detection, and morphological annotations @xcite @xcite @xcite , but none studied str between dialects. ## system overview this section presents the techniques, datasets, and the augmentation we employed in tracks a and b. ## experimental setup our experiments fine-tuned two language models for algerian and moroccan, where we used the following pre-trained models: maubmindlab/bertbase-arabertv02 (antoun et al., 2020) and ubc-nlp/arbertv2 @xcite . we employed the training data provided by the shared task, in addition to the data generated by our augmentation technique, when applied. the development data is excluded from either training or testing in the official evaluation phase, and testing is done on the shared task test set (see in the experiments of b track for the msa, no supervised fine-tuning is needed. therefore, we neither used labeled data nor augmentation. we employed average-pooling on the embeddings of the sentence tokens from the final layer in each model. then, we calculated the cosine similarity between the average embeddings of the sentences in each pair. this was done to estimate the finegrained scores for the test (or development) data provided by the shared task. the shared task considers spearman correlation score to evaluate the submitted predictions against their ground truth. ## results our approaches have achieved competitive ranks in the semrel-2024 shared task. the official results of the tracks we participated in, as well as the baselines that were introduced by @xcite , are shown in ## conclusion we presented our contributions to the semrel-2024 shared task. we targeted three arabic dialects covered by the shared task datasets, including msa, algerian, and moroccan. our approaches employed supervised and unsupervised techniques using commonly known language models, namely arbert and arabert. we augmented the training data using generative models, which enhanced the models' performance. our system ranked first (msa), fifth (moroccan), and twelfth (algerian) across the different tracks. we plan to augment additional data of moroccan and algerian using other models than what we used in this work. we will use the augmentations to experiment with both arabic mono-dialect and cross-dialect fine-tuning.