Pirates at A rabic NLU 2024: Enhancing A rabic Word Sense Disambiguation using Transformer-Based Approaches (2024)

Abstract: This paper presents a novel approach to Ara-bic Word Sense Disambiguation (WSD) lever-aging transformer-based models to tackle thecomplexities of the Arabic language. Utiliz-ing the SALMA dataset, we applied severaltechniques, including Sentence Transformerswith Siamese networks and the SetFit frame-work optimized for few-shot learning. Our ex-periments, structured around a robust evalua-tion framework, achieved a promising F1-scoreof up to 71%, securing second place in theArabicNLU 2024: The First Arabic NaturalLanguage Understanding Shared Task compe-tition. These results demonstrate the efficacyof our approach, especially in dealing with thechallenges posed by homophones, homographs,and the lack of diacritics in Arabic texts. Theproposed methods significantly outperformedtraditional WSD techniques, highlighting theirpotential to enhance the accuracy of Arabicnatural language processing applications.

## introduction arabic is recognized for its morphological richness-from diacritics to the extensive use of prefixes and suffixes. this presents significant challenges for computational linguistics. these linguistic features introduce complexities. particularly when homophones, which may differ in meaning based on diacritical marks, are standardized in academic texts @xcite . these challenges increase uncertainty, challenging the word sense disambiguation (wsd) task. wsd in arabic involves determining the contextual meaning of a word, a task complicated by homophones, homographs, contextual ambiguity, and the frequent absence of diacritics in written texts. arabic's rich inflectional and derivational morphology further complicates this task, necessitating robust and innovative computational approaches @xcite . recent literature on arabic wsd is abundant, and methodologies range from traditional knowledge-based to advanced deep-learning techniques. our study focuses on harnessing the power of transformer-based models, which have shown promise in other domains.specifically, we explore the efficacy of contextualized word embeddings, mainly through sentence transformers, enhanced by techniques such as verb sense disambiguation and arabic word sense induction based on verb preposition classes @xcite . these techniques have demonstrated promising results, particularly in contexts where traditional methods fail. drawing inspiration from contrastive learning techniques utilized in computer vision tasks such as face recognition @xcite , we adapt these approaches to arabic wsd. our methodology includes using the setfit framework @xcite , optimized for few-shot learning, which is particularly suited to the constraints of arabic language datasets in terms of size and quality. setfit's application in this context has yielded remarkable improvements with minimal data input. we were opening the possibility of using it for wsd tasks. furthermore, we evaluate a classification approach, which has proven effective against traditional wsd methods like the word expert supervised method @xcite . this approach and exploring few-shot learning techniques form the cornerstone of our comparative analysis. in summary, our study goes beyond comparing transformer-based approaches by incorporating insights from computer vision. we utilize cuttingedge, pre-trained models to adapt these advanced techniques to the specific challenges of arabic wsd. by merging these methodologies, our research aims to substantially advance arabic natural language processing. we used two dataset approaches to tackle the arabic wsd task.the first approach involved collecting data from resources similar to the salma dataset, and the second approach utilized the salma development set, which was provided by the shared task competition @xcite . ## proposed transformer-based pipeline for arabic wsd we conducted three experiments. the first experiment used the combined dataset. we used the salma-dev set as training for the second and third experiments.all models were ultimately tested on the salma test set. we used the same base model, arabertv2 (antoun et al., 2020), with three different approaches: sentence transformer with a siamese network, the setfit framework, which utilizes sentence transformer in a more optimized pipeline, and lastly, the classifica-tion approach. ## results for a comparative study, all approaches were tested on the salma test set and dev set. we used f1-score for evaluation. experiments results are shown in ## discussion our journey through the three approaches was guided by the scores we acquired during this research. starting with the arabertv2 sentence transformer, this contrastive learning approach using our combined dataset achieved a final f1-score of 48% on the salma-test set despite an 89% score on the combined dataset test set. this discrepancy indicates that the model struggled to generalize effectively. the next step involved applying the few-shot classification technique using setfit, which allowed us to utilize the salma-dev set. setfit, known for high performance with small datasets, resulted in a 50% f1-score. while this was promising, it highlighted the need for further optimization. finally, the classification approach using the arabertv2 transformer model and the salma-dev set yielded our best result with a 71% f1-score. however, this performance indicated overfitting, suggesting that the model's generalization ability could be improved. another observation is that if we compared experiment 1 and experiment 2, where the models were tested with the salma test set, we could find they had a very close performance. this indicates that the setfit few-shot approach is promising. we identified several issues that we intend to address in our future work. despite achieving the best performance with the classification approach, the scores indicate overfitting. by exploring the distribution of the datasets and its effect on model generalization, we aim to achieve better scores in the future. understanding the impact of dataset distribution on model performance is crucial, and future work will involve deeper investigations into this aspect to enhance model robustness. although setfit is optimized for intent classification rather than wsd, its application has raised several questions. for example, can we achieve better scores by using different loss functions and more tuning? ## conclusion this paper presented our system through three experiments to address arabic wsd, a task complicated by the language's complexities and limited data resources. our investigation leveraged transformer-based models, particularly the pretrained arabertv2, capable of generating contextual embeddings. we used these embeddings to tackle the arabic wsd problem. the experiments on the salma dataset yielded promising results, achieving an f1-score of up to 71%. despite the notable success of our classification approach, which secured second place in the arabicnlp 2024 competition, we identified areas requiring further exploration and improvement. these areas include optimizing model generalization to handle diverse datasets effectively and exploring more tailored loss functions. additionally, the varied results across different datasets have prompted us to consider deeper investigations into dataset distributions and their impact on model performance. future work will enhance data augmentation to generate more diverse train data to improve model generalization. additionally, we would experiment with different loss functions and hyperparameters to optimize setfit for wsd. furthermore, exploring hybrid approaches by combining multiple techniques could leverage the strengths of each approach. we also aim to incorporate strategies such as verb preposition classes @xcite .