Text Mining at S em E val-2024 Task 1: Evaluating Semantic Textual Relatedness in Low-resource Languages using Various Embedding Methods and Machine Learning Regression Models (2024)

Abstract: In this paper, I describe my submission to the SemEval-2024 contest. I tackled subtask 1 - “Semantic Textual Relatedness for African and Asian Languages”. To find the semantic relatedness of sentence pairs, I tackled this task by creating models for nine different languages. I then vectorized the text data using a variety of embedding techniques including doc2vec, tf-idf, Sentence-Transformers, Bert, Roberta, and more, and used 11 traditional machine learning techniques of the regression type for analysis and evaluation.

## introduction semantic textual relatedness (str), which involves determining the degree of semantic similarity or relatedness between two pieces of text, has emerged as a significant task within natural language processing (nlp). this task holds significant relevance and importance across various applications, including information retrieval, question answering, and summarization. by accurately measuring the semantic relatedness between sentences, we can enhance the performance of many nlp systems and improve their overall effectiveness. in this paper, we describe our participation in subtask 1-a of semeval 2024, for str of texts written in 9 languages: algerian arabic, amharic, hausa, kinyarwanda, moroccan arabic, marathi, telugu, spanish, and english. our approach to solving the task was based on a previous study that dealt with a similar sentiment classification task @xcite , and was based on a comparison of different embedding methods and then a comparison between different regression classifiers. we compared the results of each classifier with other vectors and chose the model that provided the best results on the training dataset, in favor of classifying the proximity between the sentences in the test dataset. the full description of task 1 in general and the subtasks, in particular, is given in @xcite , and the dataset is described in ousidhoum et al. (2024a). ## system overview in our study, we implemented a systematic approach to enhance the learning process of our classifier. to augment the available training data, we merged the datasets of the training and development sets. this consolidation aimed to enrich the information on which our classifier is trained. subsequently, we conducted experiments where each model was evaluated on both raw sentences and preprocessed sentences. the preprocessing steps included removing punctuation marks, numeric characters, and url addresses, and converting text to lowercase. at each stage of the learning process, we employed various text embedding methods to convert sentence pairs into vector pairs. these text embedding methods were pivotal in capturing the semantic relationships between sentences. following the generation of vector pairs, we trained a regression model to learn the semantic textual relatedness (str) label between the vector pairs. the trained model was then tasked with predicting the str level for unlabeled vector pairs present in the test set. subsequently, we performed a comparative analysis of all results and selected the best-performing models for each language under investigation. furthermore, we evaluated the performance of eleven machine learning regressors to determine their efficacy in predicting the str label. these regressors include: linear regression: a basic regression model that models the relationship between independent and dependent variables linearly. ridge regression: a regression model that uses l2 regularization to prevent overfitting. gradient boosting regressor: an ensemble learning technique that builds decision trees sequentially, each correcting the errors of the previous one. adaboost regressor: another ensemble learning method that combines multiple weak learners to create a strong learner. support vector regressor (svr): a regression algorithm that finds the hyperplane that best fits the data points while minimizing the error. svm is a supervised learning algorithm that is used for classification and regression analysis @xcite . stochastic gradient descent (sgd) regressor: a linear model trained using stochastic gradient descent. bayesian ridge regression: a regression model that is based on the bayes theorem @xcite , and assumes that features are conditionally independent given the target class, estimates the probabilities of each class and the probabilities of each feature given the class, and use it to make predictions. ## experimental setup our way of working was based on the train and dev datasets only. the goal was to train different models on the train dataset and select the best models according to the spearman rank score (according to the competition requirement) on the dev dataset. for all embedding methods(see appendix b for details), we applied the following process. in the first step, for each language, converted the sentence pairs to vectors, using different embedding methods. every method was checked twice -one with the original pair and one with a preprocessed pair. in total, for each language, we tested 5 classic embedding methods, 4 methods based on sentence-transformers, and 8 methods based on bert. that is, for each language different embedding methods were tested, once on raw text and once on pre-processed text, and for each of these methods we trained 11 regression models. we also trained additional bert models for english, spanish, moroccan arabic, and algerian arabic, so that in total we compared 3572 models (for all languages together), and at least 374 models for each language. the following tools and information sources were utilized to apply these ml methods: python 3.8 programming language @xcite , sklearn -a python library for ml methods @xcite , numpy -a python library for fast algebraic calculation @xcite , pandas -a python library for efficient data analysis @xcite , tensorflow -an open-source python library for constructing ml-dl models @xcite , and transformers -a python library for natural language processing, offering pre-trained models based on transformer architecture @xcite . hugging face -provides a platform for data scientists to access and utilize cutting-edge models (huggingface api, 2024). ## experimental results it seems that vector assignment in bert-based embedding methods was better than classical methods or sentence transformers library-based methods. this is probably due to the work that these models were massively trained on a lot of information, with the help of huge resources, and are therefore able to characterize vectors that optimally deliver the texts. also, bert models know how to characterize words with their context, and this may be a significant fact concerning the str task. in most languages, except spanish and kiryanwanda, a bert model that is multilingual was better than a bert model that was trained only on this or a similar language. this is a surprising figure as we were sure that a specific model would excel more reliably in this language. however, it seems that the models in low-resource languages are weaker and trained on less information compared to huge models from the multilingual genre. among the classical embedding methods, tf-idf seems to be the most successful method because it reaches reasonable achievements even for some of the bert models, but is still far from the best of them. the most prominent classifiers in the best models are the random forest regressor, svr regressor, and bayesian regressor. they are based on classic machine learning algorithms -random forest, support vector machine, and naive bayes which are recognized as classic classifiers but strong and good in many ml tasks. despite the well-known advantages of preprocessing methods in ml tasks, it seems that there is an overall balance between models that were quicker to preprocess their text and models that worked better on the raw text. it may be that more advanced preprocessing methods such as stemming or lemmatization will be more helpful for learning, but because in most languages it was difficult to find tools that would perform this processing of texts. ## conclusions and future research in this paper, we describe our submissions to subtask 1-a and of semeval-2024. we applied 17 embedding methods to convert text into vectors, 11 supervised machine learning methods, to predict regression of str, and did it to 9 different languages. while our study demonstrates promising outcomes across multiple languages and embedding techniques, a comprehensive error analysis reveals nuanced challenges that warrant further investigation. we observed recurrent patterns of misclassifications, particularly in contexts characterized by linguistic ambiguities, and cultural nuances, and might be affected by the prevalence of sarcasm or irony. these findings highlight the need for robust feature representation and domain-specific adaptations to enhance the accuracy and reliability of sentiment analysis models. moreover, our error analysis sheds light on the impact of preprocessing strategies on model performance, revealing a delicate balance between text normalization and the preservation of linguistic subtleties. while preprocessing techniques such as stemming or lemmatization hold promise for improving model generalization, their efficacy varies across languages and datasets, necessitating careful consideration in model development pipelines. we assume that by focusing on one or two languages, we would be able to examine the specific effect of each preprocessing method, as well as focus on the unique characteristics of each language in terms of morphological structure or methods for simplifying and decomposing words, to enable better processing and better results. there are various ideas for future research regarding the nature of twitter messages: the str task is an important task that can contribute in many fields, and this study is a milestone in my acquaintance with this task and in developing the way to do it properly.