UMBCLU at S em E val-2024 Task 1: Semantic Textual Relatedness with and without machine translation (2024)

Abstract: The aim of SemEval-2024 Task 1, “Semantic Textual Relatedness for African and Asian Languages” is to develop models for identifying semantic textual relatedness (STR) between two sentences using multiple languages (14 African and Asian languages) and settings (supervised, unsupervised, and cross-lingual). Large language models (LLMs) have shown impressive performance on several natural language understanding tasks such as multilingual machine translation (MMT), semantic similarity (STS), and encoding sentence embeddings. Using a combination of LLMs that perform well on these tasks, we developed two STR models, TranSem and FineSem, for the supervised and cross-lingual settings. We explore the effectiveness of several training methods and the usefulness of machine translation. We find that direct fine-tuning on the task is comparable to using sentence embeddings and translating to English leads to better performance for some languages. In the supervised setting, our model performance is better than the official baseline for 3 languages with the remaining 4 performing on par. In the cross-lingual setting, our model performance is better than the baseline for 3 languages (leading to 1st place for Africaans and 2nd place for Indonesian), is on par for 2 languages and performs poorly on the remaining 7 languages.

## introduction the objective of the semeval 2024 task 1 is to build and evaluate models capable of identifying relatedness between a sentence pair. sentence pairs from 14 african and asian languages belonging to 5 language groups are annotated for relatedness and released for model development. the task is divided into 3 tracks targeting different types of model training: supervised (track a), unsupervised (track b), and cross-lingual (track c). each † these authors contributed equally to this work track targets a different subset of languages. extensive details about the languages, language groups, and the data collection process are provided in the task description paper @xcite . a detailed description of the shared task, tracks, and datasets are provided in the shared task description paper @xcite . semantic relatedness helps with understanding language meaning @xcite @xcite and is useful in many areas of natural language processing such as word-sense disambiguation @xcite , machine translation @xcite and sentence representation @xcite which have numerous applications. until recently, semantic relatedness has been mostly restricted to finding word relatedness @xcite , leading to a lack of sentencerelatedness datasets. at the sentence level, relatedness has been limited to similarity, providing a restricted view of str @xcite . the current shared task aims to broaden the scope of sentence relatedness and extend it to several languages with the goal of encouraging model and resource development in these languages @xcite . recent advancements in multi-lingual translation and the availability of models for obtaining high-quality sentence embeddings allowed us to explore the effectiveness of machine translated data. using various sentence embedding models to encode data translated into english, we trained a model, transem, to find the relatedness score between sentence pairs. although the task requires the sentence pair to be from the same language, our model can handle sentences from two different languages. our second model, finesem directly fine-tunes a t5 model (t5 is already fine-tuned on the sts benchmark) on the str task using both untranslated and translated data to explore the usefulness of translation. we use both these models to evaluate languages in track a. for track c languages, we use a t5 model fine-tuned only on the english str data. for evaluating the english dataset in the cross-lingual track, we use a t5 model fine-tuned on the spanish dataset. our contributions to the str task are as follows: we 1) develop unified models for str to work with all languages. 2) participate in supervised and cross-lingual tracks. 3) explore the usefulness of machine translation. 4) explore data augmentation using machine translation. our code is publicly available at https: //github.com/dipta007/semeval24-task8 ## system overview after exploring models and datasets available in the languages we understand 1 , we realized the dearth of resources available in these languages. to leverage resources available in english, we translated the 13 non-english languages into english. assuming the translated data accurately reflects the semantic meaning of the source language, the derived relatedness value from our model for a translated sentence pair should reflect the str between the sentence pair in the source language. this section describes our machine translation process and models, transem and finesem. besides using different training strategies, these models can use both translated and untranslated data. ## results we first report our results and analysis on track a languages ( §4.1), and then on track c ( §4.2). we use the official baseline ( ousidhoum et al. (2024b) ) that used labse @xcite fine-tuned on the provided training dataset and refer to this model as baseline. ## related work in this section, we present previous research conducted in the fields of machine translation (see §5.1), sentence embedding (see §5.2) and semantic similarity (see §5.3). ## conclusion & future work we developed two different models and showed how the models performed in supervised and crossdomain training tasks in 14 languages. we explored using machine translation, sentence encoders, and sst-b style training with t5 models. our models improved over the official baseline for some of the languages. for computational purposes, we have excluded using more recent models like mistral-7b @xcite , which have outperformed most of the open-source and close-source models in various benchmarks @xcite . for future work, we intend to explore prompting for str and prompt-based llms foot_3 for translation.