INGEOTEC at S em E val-2024 Task 1: Bag of Words and Transformers (2024)

Abstract: Understanding the meaning of a written message is crucial in solving problems related to Natural Language Processing; the relatedness of two or more messages is a semantic problem tackled with supervised and unsupervised learning. This paper outlines our submissions to the Semantic Textual Relatedness (STR) challenge at SemEval 2024, which is devoted to evaluating the degree of semantic similarity and relatedness between two sentences across multiple languages. We use two main strategies in our submissions. The first approach is based on the Bag-of-Word scheme, while the second one uses pre-trained Transformers for text representation. We found some attractive results, especially in cases where different models adjust better to certain languages over others.

## introduction semantics refers to the meaning of language, including words, phrases, sentences, and overall text. understanding semantics is essential for text comprehension and communication, as it allows us to interpret the intended meaning of a message accurately. semantic relatedness measures how similar the meaning of two words or phrases is. it is based on the idea that words related in meaning tend to co-occur frequently in language or even have some causal relation connecting them. for example, cat and dog are semantically related because they refer to common household pets. measuring semantic relatedness is essential for many natural language processing tasks, such as information retrieval, question answering, and machine translation. relatedness models play a crucial role in natural language processing (nlp). these models deter-this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/ . mine the degree of similarity or relatedness between two pieces of text. one of the critical benefits of relatedness models is that they can help improve the performance of nlp applications by providing more relevant and accurate results. for example, relatedness models can be used in information retrieval to rank search results based on their relevance to the user's query. similarly, relatedness models can help identify the most relevant answer to a user's question while solving the question-answering problem. a method based on corpus-based word similarity and string similarity, as well as their order, is proposed in @xcite . for string similarity, the authors used the longest common subsequence (lcs) in three ways to weight, i.e., the work is based on measuring the shared order of words. the word mover's distance, see @xcite , reformulates the problem of comparing two sequences of words to an optimal transportation problem. it represents two sentences with its word embeddings and computes its optimal alignment using a dynamic programming solution; while it is pretty promising, it does not require sentences to be of some fixed size and works with a myriad of possible word embeddings. however, the technique was revisited by @xcite and found diverse issues that limit its effectiveness. kenter and de rijke (kenter and de rijke, 2015) have used word embeddings (word2vec) and external sources of semantic knowledge to represent text messages and meta-features. they aim to interpret proximity in the generated latent space as semantic similarity. more recently, the transformer deep neural networks have become a powerful alternative to both lexical and semantic approaches; the approach is based on a stack of encoders and decoders layers and the self-attention procedure @xcite . transformers have a high computational cost, primarily for training. the first transformer that can be pre-trained and fine-tuned to match different tasks is bert @xcite ; after bert, the high cost of training is paid once since fine-tuning needs less computational power and much less data. the interested reader should review the bert manuscript and the seminal paper about pre-training nlp models @xcite . fine-tuning bert for classification or regression tasks is straightforward, not because it is a simple architecture but due to the myriad of literature, repositories, and examples showing how to do it foot_0 . however, its usage for sentence similarity needs to produce a vector that works fine for the task, and it is not trivial to produce one with its standard matrix output. the sentence transformers @xcite use siamese networks to create effective sentence vector embeddings for tasks working with pairs of sentences, for instance, similarity search and clustering. in their research, chandrasekaran and mago @xcite have surveyed the evolution of semantic similarity methods, reviewing various nlp approaches, including traditional techniques and those found in machine learning and deep learning. they have provided a detailed study describing the strengths and weaknesses of each approach. a binary version of the relatedness tasks is as follows: given a pair of sentences u and v, predicting true if u and v are related and false otherwise. a more elaborated task is to predict a relatedness score rel(u, v) ∈ [0..1], where values near zero mean for no relation and values near 1 mean for total relatedness. the latter definition is used in the semantic text relatedness task 1 @xcite at semeval-2024, which asked for predicting relatedness scores for nine multilingual datasets; in particular, we tackled the problem as a supervised learning problem, i.e., we focused only on subtask 1 using the data for the nine languages(for more details about dataset see @xcite ). this document outlines the strategies we employed for the semantic textual relatedness (str) challenge in semeval 2024, specifically the track a for the nine languages considered. to tackle this task, we utilized two distinct approaches: a transformer method for the english and spanish languages, and an evomsa @xcite solution for the remaining languages, which include algerian arabic, amharic, hausa, kinyarwanda, marathi, moroccan arabic, and telugu. the paper is organized as follows: section 2 describes all our solutions to task 1. section 3 shows our experimental results. finally, section 4 concludes our results and findings. ## system overview nowadays, one of the most common approaches to dealing with natural language processing (nlp) problems is those transformer-based language models. however, the pre-training procedure of this kind of language model needs a vast text corpus, and therefore, it may be impossible now to train them properly in many languages. in these cases, models based on counting and computing statistics may be more robust. we used transformers for languages we know have large language models explicitly created for that language; for other datasets, we use a back-propagation optimized evomsa model for each one. ## experimental results we considered our two approaches with several expression variants for our evomsa-based approach and several models for our transformer-based approach. our model selection finds the best models using 1-pearson_correlation with k-folds crossvalidation along multilingual datasets. we selected the one+ and one-b model expressions since they demonstrated to be robust among many others coming from one+, also note that one-b works only with the training set. in particular, the transformer approach was better for spanish and english datasets. we tested with several bert, sbert, and mpnet models before selecting microsoft/mpnet-base model for english and the multilingual bert model, specifically the bert-base-multilingual-cased. 2 it is important to say that, we did not achieve outstanding results, so, further analysis cannot be done, we saw lower results in those languages less studied, and more generalized models performed better in most common languages such as english and spanish. also, in the case of less-known languages, a simpler strategy was the best such as the bag-of-words-based proposed approach. ## conclusion this manuscript describes our participation in task 1 of semantic textual relatedness (str) at se-meval 2024. we used two main approaches: a transformer-based approach and an evomsabased one. the latter has lexical and semantic representations, with variants using pre-training and fully learned from the training data. our transformer solution works better for spanish and english, while our evomsa works better for the other languages. in particular, we support low-resource languages using our evomsa without pre-trained models. our competitive results give evidence suggesting that languages with fewer resources can benefit from models that do not require an enormous corpus to be trained; this can be an alternative to large models. nevertheless, this is a very complex task, and better efforts could be made in the future.