silp_nlp at S em E val-2024 Task 1: Cross-lingual Knowledge Transfer for Mono-lingual Learning (2024)

Abstract: Our team, silp_nlp, participated in all three tracks of SemEval2024 Task 1: Semantic Textual Relatedness (STR). We created systems for a total of 29 subtasks across all tracks: nine subtasks for track A, 10 subtasks for track B, and ten subtasks for track C. To make the most of our knowledge across all subtasks, we used transformer-based pre-trained models, which are known for their strong cross-lingual transferability. For track A, we trained our model in two stages. In the first stage, we focused on multi-lingual learning from all tracks. In the second stage, we fine-tuned the model for individual tracks. For track B, we used a unigram and bigram representation with suport vector regression (SVR) and eXtreme Gradient Boosting (XGBoost) regression. For track C, we again utilized cross-lingual transferability without the use of targeted subtask data. Our work highlights the fact that knowledge gained from all subtasks can be transferred to an individual subtask if the base language model has strong cross-lingual characteristics. Our system ranked first in the Indonesian subtask of Track B (C7) and in the top three for four other subtasks.

## introduction the importance of semantic relatedness in language has been long recognized. applications include sentence representation, question answering, and text summarization @xcite . sentences can be related through either paraphrasal or entailment relations, or through broader commonalities such as shared topics, viewpoints, temporal origins, and logical connections. this shared task @xcite aims to expand the scope of significant research in natural language processing (nlp) by incorporating 14 languages. the focus of the research is on semantic similarity and has predominantly been conducted in english. the languages included in the task are afrikaans, algerian arabic, amharic, english, hausa, hindi, indonesian, kinyarwanda, marathi, moroccan arabic, modern standard arabic, punjabi, spanish, and telugu. the task requires predicting the degree of semantic textual relatedness (str) between pairs of sentences in multiple languages. the task is to rank these sentence pairs based on their level of relatedness, with scores ranging from 0 (completely unrelated) to 1 (maximally related). this task is divided into the following three tracks. track a: supervised challenge was developing a system trained on labelled datasets provided for the 11 subtasks. publicly available related datasets could be utilized, but no additional dataset could be used in this work. track b: unsupervised the challenge was developing a system without using labelled datasets to measure semantic relatedness or similarity with the text units longer than two words only. we took advantage of unigram and bigram features with svm regression and xgboost. track c: cross-lingual the challenge was to develop a system without labelled datasets in the target language and with the use of labelled dataset(s) in at least one other language (subtask). all datasets of track a other than the targeted subtask are utilized for similarity prediction in this work. our system utilizes cross-lingual learning by implementing multi-stage training methods similar to those used in @xcite , @xcite , and @xcite for track a. in the first stage, we selected pre-trained cross-lingual language models that cover the languages used in our task and fine-tuned them on a combined dataset of all subtasks in track a for five epochs. this created a model checkpoint that had knowledge of multiple languages relevant to our task. in the second stage, we fine-tuned the model checkpoint generated in the first stage for each track individually. for track c, we created a dataset by combining all the data from track a, except for the targeted subtask, and fine-tuned it with the language model in a supervised manner. we used unigram and bigram bag-of-words representation with svm-based regression (svr) and xgboost for each subtask in track b. our team achieved the best model for the telugu and marathi subtasks of track a by using the muril large model @xcite . in the first stage, we fine-tuned the model for all three tracks (english, telugu, and marathi) and then fine-tuned the model checkpoints for telugu and marathi. for all track a languages, we fine-tuned xlm-r for subtasks in the 1st stage, and we fine-tuned the checkpoint generated in stage one for all monolingual tracks in the 2nd stage. for track b, only monogram or bigram representation is allowed for supervised training. we obtained comparable results using both unigram and bigram representations in combination with svr and xgboost. in track c, we used all training data from track a except for the current subtask data since the use of the same language data was not allowed in track c. we adopted a cross-lingual transfer approach, where muril gave the best result for the hindi subtask, while xlm-r predicted the best results for the other subtasks. the results for each subtask are presented in ## related work the semantic textual similarity (str) task 2015 @xcite had three subtasks. the findings showed that the umbc pairingwords system achieved the best score by semantically differentiating distributionally similar terms @xcite . in the subsequent str task @xcite , there are seven tasks that concentrate on multilingual and cross-lingual pairs. additionally, one sub-track will delve into mt quality estimation data. the team ecnu @xcite achieved the highest score using ensembles of well-performing featureengineered models with deep learning methods. these models used random forest (rf), gradient boosting (gb), and xgboost (xgb) regression methods. however, statistical and machine learning models were not the best, as transformer-based models gained attention after @xcite . these models are pre-trained on large amounts of data and fine-tuned for various downstream tasks. researchers have created the sentence transformer @xcite architecture for finding similar sentences. it is useful when given multiple sentences corresponding to a sentence, and we need to find the most similar one. however, fine-tuning the sentence transformer with the downstream task requires proper alignment between the dataset on which the sentence transformer is pretrained and the dataset of the downstream task. in this task, there are multiple subtasks associated with multiple languages. therefore, motivated by the performance of cross-lingual transformer-based models, we have used transformer-based language models that have strong cross-lingual transferability. it has been seen that cross-lingual transferability has advantages in various nlp tasks @xcite . ## data for track a. the detailed methodology for each track is explained in the following subsections. for tracks a and c, we combined both sentences of an example data with a special token of lm (</@xmath0, and [sep] for muril) in the preprocessing stage. model tokenizer tokenizes the combined sentence into tokens and generates token ids and attention masks. for other subtasks, we utilized xlm-r and muril with the two-stage training approach. ## results and analysis the table below shows the pearson score with official rank for track a, track b and track c, along with the baseline score. please refer to track a: two-stage muril setup achieved the best scores for telugu and marathi subtasks, while two-stage xlm-r setup achieved the best score for all other subtasks.