M ason T igers at S em E val-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness (2024)

Abstract: This paper presents the MasonTigers’ entry to the SemEval-2024 Task 1 - Semantic Textual Relatedness. The task encompasses supervised (Track A), unsupervised (Track B), and cross-lingual (Track C) approaches to semantic textual relatedness across 14 languages. MasonTigers stands out as one of the two teams who participated in all languages across the three tracks. Our approaches achieved rankings ranging from 11th to 21st in Track A, from 1st to 8th in Track B, and from 5th to 12th in Track C. Adhering to the task-specific constraints, our best performing approaches utilize an ensemble of statistical machine learning approaches combined with language-specific BERT based models and sentence transformers.

## introduction in this modern era of information retrieval and nlp, understanding semantic relatedness is fundamental for refining and optimizing diverse applications. semantic relatedness refers to the degree of similarity and cohesion @xcite in meaning between two words, phrases, or sentences. semantic relatedness allows systems to grasp the contextual and conceptual connections between words or expressions. various nlp tasks and applications can benefit from modeling semantic relatedness such as question answering @xcite , knowledge transfer @xcite , text summarization @xcite , machine translation @xcite , and content recommendation @xcite . while significant research has been conducted on semantic relatedness in english, more recently the interest in semantic relatedness in other languages has been steadily growing @xcite . this reflects an increasing awareness of the need for developing models to languages english other than english. nlp is evolving rapidly and we have been witnessing the emergence of languagespecific transformer, the release of datasets for downstream tasks in diverse languages, and the development of multilingual models designed to handle linguistic diversity. semeval-2024 task 1 -semantic textual relatedness @xcite aims to determine the semantic textual relatedness (str) of sentence pairs across 14 diverse languages. track a focuses on nine languages (algerian arabic, amharic, english, hausa, kinyarwanda, marathi, moroccan arabic, spanish, telugu) using a supervised approach where systems are trained on labeled training datasets. track b adopts an unsupervised approach, prohibiting the use of labeled data to indicate similarity between text units exceeding two words. this track encompasses 12 languages (afrikaans, algerian arabic, amharic, english, hausa, hindi, indonesian, kinyarwanda, modern standard arabic, moroccan arabic punjabi, and spanish). track c involves cross-lingual analysis across the 12 aforementioned languages. participants in this track must utilize labeled training data from another track for at least one language, excluding the target language. evaluation across all three tracks involves using spearman correlation between predicted similarity scores and human-annotated gold scores. we conduct distinct experiments for each track using statistical machine learning approaches along with the embeddings generated by transformer based models. ## related work understanding the level of semantic relatedness between two languages has been regarded as essential for grasping their meaning. notable studies on the topic including @xcite @xcite @xcite ; @xcite and @xcite have introduced datasets like sts, mrpc, and lisent. these datasets have been piv-otal in advancing research in tasks such as text summarization and plagiarism detection. finding semantic relatedness and semantic similarity, as well as determining sentence pair similarity using existing datasets or paired annotation, are integral in understanding the nuances of language comprehension. previous studies describe how words and sentences are perceived to convey similar meanings (halliday and @xcite @xcite @xcite . methodologies like paired comparison represent the most straightforward type of comparative annotations @xcite , @xcite . best-worst scaling (bws) (louviere and woodworth, 1991) a comparative annotation schema, offer insights into methods for evaluating relatedness through pairwise comparisons. the utilization of these methods aids in generating ordinal rankings of items based on their semantic relatedness. @xcite highlight the effectiveness of such techniques, emphasizing the importance of reliable scoring mechanisms derived from comparative annotations for understanding the intricacies of semantic relatedness in natural language processing tasks. ## data the shared task comprises three tracks: supervised, unsupervised, and cross-lingual. the dataset @xcite is comprised of two columns: the initial column, labeled "text," containing two full sentences separated by a special character, and the second column, labeled as "score", which includes degree of semantic textual relatedness for the corresponding pair of sentences. in the supervised track (track a), there are 9 languages, and for each language, train, dev, and test sets are provided. the specifics of the dataset for this track can be found in finally, in the cross-lingual track (track c), there are 12 languages and for all the languages dev and test set is provided and they are same as the unsupervised track. here the training dataset is not provided. hence, for each individual language of this track, we select 5 languages from supervised track (different from the target language) and merge training data of those five languages to create the training dataset for each of the languages of crosslingual track. the details of the dataset of this track is available in ## experiments we use statistical machine learning along with language specific bert-based models to find the sentence embeddings and predict relatedness between pair of sentences. additionally, we use sentence transformers for the supervised track. our experiments are described in detail in the next sections. ## results for all the tracks, ensemble of the predictions prove helpful in terms of achieving better spearman correlation. for track a sentence transformer labse along with linear regression performs the best among the eight combinations for all the languages. then the weighted ensemble improves the result 1%to 3% in development phase and 1% -2% in evaluation phase -depending on the languages. for english this method performs the best in terms of ranking with 11 th rank while the worst for moroccan arabic with 21 th rank. on test spearman correlation, english is the best securing 0.84 and kinyarwanda is the worst with 0.37. detailed results are shown in for track b, embedding generated by language specific bert based models provide the best result among the three combinations for all the languages. then the average ensemble improves the result 0% -to 3% in development phase and 0% -2% in evaluation phase -depending on the languages. for kinyarwanda this method performs the best in terms of ranking with 1 st rank while the worst for english with 8 th rank. on test spearman correlation, english is the best securing 0.77 and punjabi is the worst with 0.02. detailed result is shown in for track c embedding generated by language specific (unrelated to target language) bert based models provide the best result among the six combinations for all the languages. then the average ensemble improves the result 0% -to 2% in both development and evaluation phases depending on the languages. for punjabi this method performs the best in terms of ranking with 5 th rank while the worst for hausa and kinyarwanda with 12 th rank. on test spearman correlation, spanish is the best securing 0.56 and punjabi is the worst with 0.02. detailed result is shown in ## error analysis for track a, algerian arabic, moroccan arabic and spanish test spearman correlation coefficient decreases in the evaluation phase. this happens because the dev set was around 7.5%-9% and the test set is around 39% -46% size of the train set. for track b, amount of dev data was only 20 for afrikaans which is the reason of a very big difference between the result of development and evaluation phase. algerian arabic, amharic, modern standard arabic, moroccan arabic have a very small amount of dev data (less than 100) which is reason of decreased spearman correlation coefficient in the evaluation phase. hindi also faces the same issue but as it had more dev data the test spearman correlation coefficient is only 4% less than the development period. for track c, algerian arabic, indonesian, kinyarwanda, modern standard arabic faced bigger drop of the spearman correlation coefficient from the development phases. the main issue here is the bert based models that doesn't know the target languages generate the embeddings that are not as good as what we observed in unsupervised track for the models with the knowledge of target language. also the diversity of the train and test data make it more challenging to score better spearman correlation coefficient. in addition, due to the unavailability of the text label, only the ensemble performance of spanish language for all the tracks are shown. regarding the result of the punjabi language in the both unsupervised and cross-lingual track, it was the most challenging language where the provide baseline was less than zero. though our system achieves 0.02 spearman correlation coefficient for for this language, the ranking is quite figure 2: track b (comparison with gold semantic textual relatedness) figure 3: track c (comparison with gold semantic textual relatedness) moreover, elasticnet and linear regression exhibit limitations as assumption of linearity may not align with the intricate and nonlinear relationships inherent in the textual data. the issue of dimensionality poses a challenge, especially when dealing with a large number of features. the difference between the gold and predicted semantic relatedness scores for the three tracks are shown in figure 1 , figure 2 , and figure 3 . ## conclusion we experimented with various methodologies on the dataset provided by the organizers, including statistical machine learning approaches, transformer-based models, language-specific berts, and sentence bert. in the supervised task (track a), with no restrictions on the model or data, we utilized the available training dataset. conversely, the unsupervised task (track b), lacking training data, presented challenges, leading us to use language-specific berts and statistical machine learning approaches. the cross-lingual track (track c) imposed more stringent restrictions, requiring us to use training data from other languages in track a, excluding the target language. in addition to statistical ml models, we integrated language-specific berts closely aligned with the geography and culture of the target language, as the use of llms was constrained due to unknown training data. we show that our ensemble approach exhibited superior performance compared to individual model experiments. however, the task's inherent difficulty became evident in instances where relatively small datasets presented challenges for effective model learning. semantic textual relatedness tasks face challenges like subjectivity, context dependency, and ambiguity due to multiple meanings and cultural differences. limited data, domain specificity, short texts, and biases hinder accuracy. ongoing research is crucial to address these limitations and improve model accuracy.