M ai NLP at S em E val-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness (2024)

Abstract: This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences from the same languages. For cross-lingual approach we developed a set of linguistics-inspired models trained with several task-specific strategies. We 1) utilize language vectors for selection of donor languages; 2) investigate the multi-source approach for training; 3) use transliteration of non-latin script to study impact of “script gap”; 4) opt machine translation for data augmentation. We additionally compare the performance of XLM-RoBERTa and Furina with the same training strategy. Our submission achieved the first place in the C8 (Kinyarwanda) test.

## introduction the task of semantic textual relatedness (str) has a long-standing tradition in nlp (e.g., @xcite . it consists of predicting a score that reflects the closeness in semantic meaning between two given sentences. for example, consider the following examples extracted from the actual shared task data @xcite shown in figure 1 . for english, the annotators scored the first pair higher than the second sentence pair. similarly, for afrikaans the annotators scored the first example higher than the second one. as further described in @xcite , all sentence pairs were annotated manually in a pairwise fashion to obtain semantic textual relatedness (str) scores between 0 (completely unrelated) and 1 (maximally related). while previous work has largely focused on english, the semeval-2024 shared task 1 @xcite figure 1 : examples from the dev sets for semantic textual relatedness (str). eng: english, afr: afrikaans. et al., 2024b) aims to extend the language coverage. it proposes datasets to evaluate the relatedness of sentence pairs for a total of 14 languages, including low-resource tail languages such as kinyarwanda (kin) or marathi (mar) @xcite (see §2.1). the shared task includes three subtracks, each with a focus on supervised, unsupervised and cross-lingual str, respectively. in this paper, we focus on track c, cross-lingual str. in this track, the goal is to develop a system to predict str scores without access to any labeled data for the target language (importantly, also no target development data). that is, track c requires the development of a regression model for 12 target languages, without relying on any labeled datasets in the target language (or pre-trained language model fine-tuned on other str tasks). instead the cross-lingual task allows to utilize training datasets from at least one other language from the other tracks (which includes training data of up to 9 languages). returning to our running example in figure 1 , the task is to develop a system for example for afrikaans as target by transferring knowledge from one or more source languages (which may include english). previous work on multilingual nlp has illustrated the curse of multilinguality @xcite , that is, diminishing returns for training a single system on many languages due to language interference. this shared task has a focus on lowresource languages and languages typologically distant to english, a setup in where cross-lingual transfer has shown to be particularly challenging @xcite . motivated by these two aspects, we set out to study the use of fewer but more relevant source languages for a given target language. more specifically, we aim to find good "donor language(s)" @xcite and compare those to baselines that either only use english, or a multi-source model trained on all source languages (except the target). we aim to answer the following research questions: rq1 to what extent does knowledge transfer from source languages improve str models? rq2 do multilingual str models exhibit language interference @xcite , i.e., performance drops when training data from heterogeneous languages are combined? rq3 to what extent do script differences play a role in str ("script gap"), and can we narrow the script gap by using a foundation model specialized to align transliterated data and data written in different scripts? rq4 can we further improve the transfer performance by relying on machine translation to augment existing training data? to study rq1, we make use of typological information available in language vectors. for rq2, we opt for a multi-source approach, that combines the training data for all languages (except the target). to study the impact of scripts (rq3), we make use of transliteration, and further compare a bert-based model to furina @xcite , a recently proposed language model that aims to better align languages across scripts. finally for rq4, we investigate the use of machine translation (mt) for data augmentation. we apply ours methods to 12 target languages in track c. the specific details about languages are presented in §2.1. ## methods we opt for two roberta-based @xcite models for the regression task trained with a mean-squared error (mse) loss. more specifically, we use the xlm-roberta base model, and furina @xcite , which is a xlm-r derivative based on glot-500 @xcite , further detailed below. we adopt a multi-source approach that involves individually fine-tuning a model for each target language in track c. this fine-tuning process utilizes the training datasets from all languages available in track a, explicitly excluding the dataset of the test language itself. for baseline comparisons, we use xlm-roberta @xcite and furina @xcite models fine-tuned solely on english datasets. labse. the organizers provide cross-lingual baselines for each target language by fine-tuning language-agnostic bert sentence embeddings (labse) @xcite , which supports 109 languages. labse was pre-trained using translation language modeling (tlm) @xcite , which included bilingual translation sentence pairs for training. the bilingual corpus is constructed from web pages using a bitext mining system, filtered by a pre-trained contrastive dataselection scoring model, and manually curated to create a high-quality collection of 6 billion translation pairs. out of those, labse has been exposed to different amounts of parallel data (eng-xxx) from semreleval languages. the largest amount of parallel text involves spanish with over 375m sentence pairs (eng-esp), followed by indonesian with over 250m sentence pairs (eng-ind), followed by hindi and arabic (eng-{hin, arb}) with over 125m language pairs. all other languages (afr, pan, amh, haus, tel, kin, mar) appear in the tlm training corpus with less than 125m sentence pairs. ## experimental setup the detailed setting are listed in appendix a. as baseline, we exclusively train a model on the english dataset (xlm-r eng , furina eng ) and assess its performance across all target languages. subsequently, for each target language, we fine-tune a multi-source model: if the target language is not within the 9 training datasets, we train on the union of all @xmath0. otherwise we train a multi-source model on n -1 = 8 source languages, excluding the target (xlm-r ms-all , furina ms-all ). following this, we explore whether it is helpful to prune certain languages, retaining only english and the two closest to the target languages according to lang2vec (littell et al., 2017) foot_3 as source languages (xlm-r l2v , furina l2v ). due to the reduction in the training set, which significantly decreased the size of the data, we attempted to expand the dataset through crosstranslation (xlm-r l2v-aug , furina l2v-aug ; cf. §3.2). ## results and discussion our main results are presented in single-source versus multi-source transfer. we first compare the performance of a zero-shot str model trained on english (xlm-r eng , furina eng ) against a multi-source model trained on the concatenation of all available languages from track a (xlm-r ms-all , furina ms-all ). our results reveal that knowledge transfer from multiple source languages (rq1) improves str models, affirming the potential of multi-source training to enhance crosslingual capabilities. on average, both ms-all models outperform their single-source counterparts by 0.02 and 0.09 respectively. this is expected since we exclude english from the average result (avg). bold: best result for each language. languages not covered by all l2v features are excluded from the average (eng, pan, arq, ind, kin). for our knn-variants we opt for @xmath1. the multi-source training dataset is with 15,123 instances almost three times larger than the english dataset with 5,500 instances (cf. transfer from language families. after showing that models trained on all languages outperform the single-source baseline, we now investigate the effect of training on languages from the same family as source languages. here we experiment with two multi-source models specialized only on indo-european and afro-asiatic languages respectively (ms-fam). importantly, for each target language we train a multi-source model on all other languages in the same language family. foot_4 on indo-european languages, we find that xlm-r ms-fam and furina ms-fam yield similar results with much less training data (i.e., 4,913 fewer instances belonging to other language families). for spanish, our models show performance gains of +0.8 and +0.13 for xlm-r and furina respectively, when compared to models trained on all languages. this underscores the presence of language interference @xcite in multilingual str models when the training data from dissimilar languages are combined (rq2). on afro-asiatic languages, we observe average performance drops of -0.09 and -0.06 for xlm-r and furina when moving from ms-all to ms-fam. we hypothesize that this can be attributed to the amount of training data available. in fact, there are 28% fewer training instances for all afro-asiatic languages than for english (5,500). transfer from nearest language neighbors. we now investigate the transfer performance when training str models on their two closest languages according to cosine similarity of language cell state vectors, i.e. learned language vectors presented in @xcite . as mentioned earlier, we add english due to its large scale as a third training language. our submitted system, furina eng+esp+hau , is trained on the two closest training languages of kinyarwanda (kin) and has been ranked first place on the shared task leaderboard. applying the same approach for each test language (xlm-r knn , furina knn ) shows mixed results. this indicates that the strong performance on kin can be attributed to the fact that, contrary to xlm-r, kin has been seen by furina during pretraining. single-source transfer results. we now select the most similar source languages based on different typological features obtained from the lang2vec (l2v) library. we obtain l2v vectors for phonology (pho), syntax (syn), inventory (inv), family (fam), geography (geo) and learned (lrn) features. ## conclusion in this paper, we investigate source language selection for cross-lingual transfer for semantic textual relatedness (str). we evaluate three different language selection strategies: single-source, multisource transfer and transfer from english and two nearest language neighbors. we find that the transfer performance crucially depends on the size of the training dataset and the linguistic proximity to the test language. we further show that script differences cause high variance transfer performance and mt-based data augmentation can lead to shifts in label semantics. fine-tuning furina on eng, esp, and hau, we achieve first place in the semeval-2024 task 1, track c8 (kin).