OZ emi at S em E val-2024 Task 1: A Simplistic Approach to Textual Relatedness Evaluation Using Transformers and Machine Translation (2024)

Abstract: In this system paper for SemEval-2024 Task 1 subtask A, we present our approach to evaluating the semantic relatedness of sentence pairs in nine languages. We use a mix of statistical methods combined with fine-tuned BERT transformer models for English and use the same model and machine-translated data for the other languages. This simplistic approach shows consistently reliable scores and achieves above-average rank in all languages.

## introduction semeval 2024 task 1 @xcite calls for assigning scores indicating semantic textual relatedness (str) of sentence pairs in 14 different languages. we participate in track a, which is the supervised subtask for systems that have been trained using the provided labeled datasets @xcite . there are data in algerian arabic, amharic, english, hausa, kinyarwanda, marathi, moroccan arabic, spanish, and telugu for track a and we provide a solution for all 9 languages. the labeled data has been manually annotated for relatedness using a comprehensive annotation framework @xcite . a large portion of previous work in str has been conducted for english-language data. this task does include english, but the focus is on lower-resourced languages @xcite . str is a crucial component in information retrieval, summarization, and question answering, as well as in developing large language models (llms). the lack of str or similar nlp resources for low-resource languages means progress is often much slower in related research such as the development of llms too making the progress achieved through this task societally highly impactful by providing new tools and datasets for language where nlp resources are lacking @xcite . our methodology uses both traditional tf-idf vectorization and transformer models like bert @xcite fine-tuned for semantic relatedness tasks. we leverage the high availability of resources that exist for english to fine-tune a bert model that we then use on machine-translated versions of the datasets for the other languages (except for spanish where a multilingual bert model yielded better results than with machine translating the data). this approach seems to capture both lexical patterns and deeper semantic relationships, making it effective for linguistically diverse datasets, and cost-effective because there is no need to manually annotate more than one dataset (language). it is therefore an alternative approach to creating language-specific models. although our approach is simplistic, it has the upside of working reasonably well for any low-resource language that has some machine translation or parallel language data resources. ## background in semeval-2024 task 1, the dataset was adapted from the str-2022 dataset @xcite . the str-2022 dataset contains 5,500 english sentence pairs that were manually annotated using a comparative annotation framework, yielding finegrained scores ranging from 0 to 1 (maximally unrelated to maximally related). the dataset was constructed by sampling sentences from various sources to capture a wide range of text characteristics such as sentence structure, formality, and grammaticality. the sources include datasets on formality @xcite , book reviews @xcite , paraphrases @xcite , natural language inference @xcite , semantic textual similarity @xcite , stance @xcite , and text simplification @xcite . the corresponding datasets for the other languages are much smaller and consist of roughly 1000 sentence pairs each with minor variations in size. semantic relatedness and semantic similarity are closely related concepts in natural language processing (nlp), however, the terms are not interchangeable. semantic similarity is a narrower definition that only takes term similarity into account (e.g. fork is similar to knife), whereas relatedness in addition to similarity can include terms or concepts that are related beyond hyponymic relationships such as fork being related to eating) @xcite . this task focuses on the broader concept of relatedness but utilizes more narrowly defined datasets based on similarity as well in the construction of the datasets. in recent years the development of nlp resources for low-resource languages has been speeding up, but there are still large discrepancies in what types of tools, models, and resources exist for languages other than english @xcite . there are also significant differences in the resources available among low-resource languages and what being a low-resource language entails @xcite . for most of the languages in this task, there are at least some models and tools (see e.g deode et al., 2023) but a handful of research groups working on a language is quite different from nearly all research groups in the world working on producing models and tools for a language (english). when there is a need for more data, often data augmentation methods are used to increase data points. machine translation is an established method of data augmentation, particularly with low-resource languages where it might not be possible to use language-specific models @xcite . ## system overview our choice of methodology was shaped by pedagogical considerations as well as technical. as we participated in this task as part of an undergraduate senior research seminar in computational methods, we purposely started with the simplest most readily available tools progressing towards more advanced methods. along the way, we compared the results and progress at each step in an attempt to better understand how each of the specific nlp tools worked and how accurate their output was when used on real projects such as this dataset. the main strategy of our system is integrating classic nlp methods, such as the dice score and tf-idf, with advanced deep learning techniques like bert models, to determine semantic relatedness between sentence pairs. firstly, our system imports a csv dataset that contains pairs of english sentences (separated by "\n"), each paired with a relatedness score ranging from 0 to 1. then, to assess semantic relatedness, the system adopts several basic nlp techniques, including spacy's linguistic features for efficient text processing, tf-idf for calculating word importance in sentences, spacy similarity and cosine similarity for measuring sentence similarity, and fine-tuned bert models for leveraging contextually rich semantic analysis @xcite . these techniques collectively contribute to a robust evaluation of semantic relatedness against the given scores. we tried early on to adopt the same approach to the non-english languages with language-specific transformer-based similarity and relatedness models, but the languagespecific models yielded much lower evaluation scores than what the english model achieved with machine-translated versions of the non-english datasets. we used the google translate api to translate the datasets into english to maintain consistency in analysis. compared to other translation apis such as deepl, for this task, google translate seemed to produce better translations, perhaps because of how it favors more common words over context thus being more suited for str and/or sts tasks (see e.g. @xcite . participating in the semantic relatedness task using the hybrid strategy allows for a comprehensive exploration of the system's performance and methodology. through a detailed analysis, you can assess the effectiveness of traditional nlp methods, including tf-idf and spacy's linguistic features, in comparison to more advanced deep learning techniques like bert. evaluating the impact of contextual embeddings from fine-tuned bert models provides insights into how well the model captures nuanced semantic relationships. the inclusion of google translate for non-english languages offers an opportunity to examine the system's ability to maintain consistency across languages. assessing the generalization capability, scalability, and efficiency of the system provides a holistic understanding of its applicability to diverse datasets and real-world scenarios. through this participation, we can uncover strengths, weaknesses, and potential areas for improvement, guiding future research directions and refining the hybrid strategy for en-hanced semantic relatedness evaluation across languages and varied linguistic contexts. in particular, this approach shows that it is possible to achieve reasonable accuracies by leveraging the prevalence of tools and models designed for english with lowresource languages. our code is available on github foot_0 . at the beginning stage of the experiment, we undertook an examination of several readily implementable models on the english baseline dataset and compared the predicted scores with humanlabeled scores through pearson correlation scores. in the initial english baseline model, we included the spacy similarity model 4 , cosine vector similarity, and fine-tuned-bert models 5 . for the spacy similarity, we directly applied it to the training dataset, yielding a result of 0.34 (pearson). in the case of cosine similarity, we tried out two methods of word embedding: upon comparing these two word-embedding methods, the pearson correlation results did not reveal a significant difference. therefore, we selected the binary occurrence method as the cosine vector similarity, which achieved a score of 0.61 as indicated in the final component of the english baseline is the application of the fine-tuned bert model to compute semantic relatedness with the (unfinetuned) clinicalbertsimilarity 5 and webbertsimilarity 5 models and a batch size of 10 for both. the creators of the model claim that the "project contains an interface to fine-tuned, bert-based semantic text similarity models. it modifies pytorchtransformers by abstracting away all the research benchmarking code for ease of real-world applicability" 5 . this proved to be the most successful approach with a result of 0.8 for english. although the task in question is about semantic relatedness, since many of the datasets involved in the creation of the datasets come from similarity data. additionally, as similarity can be considered a subtype of relatedness, the use of similarity models seemed logical due to their wider availability compared to relatedness models. after establishing the english baseline, we evaluated several multilingual and language-specific bert-based similarity models to assess textual relatedness (or similarity) across other language training datasets including the sbert model for telugu @xcite , sentence-bert @xcite , biolord-2023 @xcite , etc. however, the results were suboptimal, which is surprising since previous work has shown that sentence transformers show significant improvements to semantic similarity tasks, particularly cross-lingual tasks @xcite . given the significantly better performance of the english baseline, we decided to translate all language datasets into english before applying the relatedness prediction models. in the case of spanish we found that using distiluse-base-multilingualcased-v1 @xcite produced higher accuracies than the translation approach, and thus spanish is the only language we did not translate to english. when introducing the translation tools, we explored two approaches: utilizing a translation model (machine translation) and implementing google translate. 1. machine translation: in the machine translation method, we applied m2m100 @xcite as the translation model. the model can directly translate between the 9,900 directions of 100 languages. despite the relatively high performance claimed by the m2m100 model as described by @xcite , the results after the translation process are less than 0.5 for all languages except spanish, where it achieved a result of 0.67. in contrast, the google translate api demonstrated better performance during the training process with the english baseline model (detailed results are listed in our multilayered approach mirrors that of jeyaraj and kasthurirathna (2021) although ours is a much simpler setup. ## results our rankings show that our approach is nowhere near the state-of-the-art, but it is still a reliable option when more language-specific approaches are unavailable as is often the case with moderately low-resource languages. out team ranked in the middle of the pack for most languages, but in the top third for english, marathi, and spanish, and the bottom for both arabic dialects, which was expected. the rankings, scores, and models used for each submission can be seen in ## conclusions to sum up, we first focused on english to have a good solution with fine-tuned bert, and then we applied that solution to other languages by translating the sentences into english using machine translation. since our english solution is reasonably good (rank 10/34, official score of .81), the application of the solution worked much better than using multilingual models in many languages including amharic, marathi, and telugu for which there exist language-specific semantic similarity models. we speculate that the reason the mt+english model worked better than the language-specific relatedness models is due to the higher quality and more diverse training data for the english model(s) as well as machine translation simplifying words to the most commonly used ones, artificially making similar sentences more similar. the importance of an accurate machine translation can be seen in the failure of our approach with the arabic dialects in particular. google translate does not have specific translators for moroccan or algerian arabic, instead, we had to rely on general arabic. this likely produced much lower quality translations obfuscating the semantic links between the sentence pairs making it difficult for the english model to accurately judge relatedness. this issue was further exacerbated by the fact that no one on our team speaks any of the languages in the task besides english, which made manual evaluations of the mt output difficult. darja and darija are the names for algerian and moroccan arabic respectively, and they are collectively known as maghrebi arabic. due to its roots in berber languages, there are notable distinctions between maghrebi arabic and standard arabic, and using the latter for these two dialects may yield a suboptimal result. curiously, a similar issue occurred with spanish. spanish is much more closely related to english than the other languages in subtask a, and therefore we expected our approach to get a fairly high score similar to english, especially considering the current state of machine translation between english and spanish. however, it seems that translation of spanish into english affects the semantic relations of the original sentences, which might be one of the main reasons causing the very low scores and making us choose the multilingual model for spanish rather than the machine-translated one. we hypothesize that one of the reasons that google translate worked so well on the lowresource languages most dissimilar from english might be because smaller training datasets for mt would force the translation to use less context and instead increase the reliance on individual lexical items leading to sentence pairs with high relatedness becoming more similar via translation. for languages with better mt models, it is conceivable that the better translations work against this approach as it might make the sentence pairs less similar as reflected by the higher scores for spanish using multilingual models, and the very low scores for both arabic dialects. in future work, it might be worthwhile to use mixed methods starting with language-specific models and then expanding to incorporate machine translation and larger models developed for, e.g., english.