rematchka at A rabic NLU 2024: Evaluating Large Language Models for A rabic Word Sense and Location Sense Disambiguation (2024)

Abstract: Natural Language Understanding (NLU) plays a vital role in Natural Language Processing (NLP) by facilitating semantic interactions. Arabic, with its diverse morphology, poses a challenge as it allows multiple interpretations of words, leading to potential misunderstandings and errors in NLP applications. In this paper, we present our approach for tackling Arabic NLU shared tasks for word sense disambiguation (WSD) and location mention disambiguation (LMD). Various approaches have been investigated from zero-shot inference of large language models (LLMs) to fine-tuning of pre-trained language models (PLMs). The best approach achieved 57% on WSD task ranking third place, while for the LMD task, our best systems achieved 94% MRR@1 ranking first place.

## introduction natural language understanding (nlu) is a crucial field in language processing that aims to comprehend and interpret human language. word sense disambiguation (wsd) plays a crucial role in resolving multiple meanings and nuances of words, but applying wsd to the arabic language presents unique challenges due to its rich morphology, dialectal variations across different regions, and polysemy. a special case within wsd is location mention disambiguation (lmd), which involves disambiguating location references in arabic texts considering diverse geographical, historical, and cultural contexts. developing effective nlu techniques for arabic, particularly for wsd and lmd tasks, is of great significance for various applications such as information retrieval, machine translation, and natural language understanding systems. however, there are not enough good linguistic resources and datasets specifically designed for these tasks in arabic, making it difficult to train and test models, affecting their accuracy and reliability. this paper outlines our solution to the arabicnlu shared-task @xcite , which involves two tasks. the first task is word sense disambiguation (wsd), where the objective is to determine the correct meaning of a target word within a given context by selecting from a set of candidate senses (i.e., glosses or definitions) associated with the target word. the second task is location mention disambiguation (lmd). in lmd, the goal is to match each location mention within a post or tweet to the correct toponym from a geo-positioning database that contains a list of toponyms. this matching process aims to find the toponym in the database that accurately represents the mentioned location. to address the wsd challenge, we conducted experiments with large language models (llms) in a zero-shot setting. in this approach, we have instructed the model to select the appropriate sense from a list of senses given the context, and target word. additionally, we explored the effectiveness of fine-tuning models like marbert and arabert for improved performance in wsd. for the lmd task, we have experimented with llms in a zero-shot setting and in the context of the retrieval-augmented generation (rag) framework. in the zero-shot setting, we provided instructions to the model to translate the location and provide the corresponding country name as a means to obtain the accurate toponym from the geopy framework. in the rag framework, we maintained a record of the context, location, country, and the valid toponyms that were available. retrieving the toponym involved evaluating the similarity score between the new context and location, and comparing it with the knowledge base to select the highest-scoring match. the experiments aimed to determine which approach yielded better results and provided insights into the effectiveness of llms in solving wsd and lmd challenges in arabic natural language understanding. the rest of the paper goes as follows: section 3 gives an overview of the dataset, section 4 discusses the proposed methods, section 5 shows experimental results, and section 6 concludes the paper. ## related work this section of the paper provides a literature review on arabic word sense disambiguation and location mention disambiguation. it discusses the importance of these tasks in accurate and contextaware natural language processing in arabic. the review focuses on existing approaches, resources, and evaluation methodologies employed in the field. @xcite proposed arabglossbert dataset, which consists of 167k context-gloss pairs collected from arabic dictionaries, is commonly used but relatively small. the authors present an enriched version of the dataset achieved through machine back-translation, increasing its size to 352k pairs. they evaluate the impact of augmentation using various data configurations to fine-tune bert for the target sense verification (tsv) task. the accuracy ranges from 78% to 84% across different data configurations, with some improvements observed for specific parts-of-speech (pos) in certain experiments. @xcite focuses on fine-tuning bert models for arabic wsd. the authors constructed a dataset of labeled arabic context-gloss pairs, extracted from the arabic ontology and a lexicographic database. each pair was labeled as true or false, and target words in each context were identified and annotated. three pretrained arabic bert models were then fine-tuned using this dataset. the authors also experimented with different supervised signals to emphasize target words in context. the experiments yielded promising results, achieving an accuracy of 84%, even when dealing with a large set of senses in the experiment. @xcite introduces a new dataset for arabic wsd and proposes a novel approach using bert. the proposed wsd system outperforms existing methods, achieving an approximate f1-score of 96%. the effectiveness of wsd is demonstrated in a case study involving sentiment analysis as a downstream task. @xcite addresses the challenges of wsd in arabic due to limited resources and semantic sparsity. the authors propose wsdtn, a manually annotated corpus consisting of 27,530 sentences collected from diverse sources. each sentence includes a target word and its corresponding sense. they also present a transformer-based model for disambiguating new words and evaluate the corpus's performance. the baseline approach achieves an accuracy of approximately 90%. @xcite presents a hybrid approach for arabic word sense disambiguation (awsd) by combining a single-layer convolutional neural network (cnn) with contextual representation using bert. the proposed approach utilizes a concatenation of bert models as word embeddings to capture both target and context representations. experimental results demonstrate that the proposed model surpasses state-of-the-art approaches, achieving an accuracy of 96.42% on the arabic wordnet dataset, thereby improving the performance of wsd in arabic languages. ## data in subtask 1, the development dataset provided by the organizers is a subset of the salma corpus @xcite . the training dataset consisted of 100 sentences, while the test set comprised 1,340 sentences. as for subtask 2, the training dataset used in this task was derived from the idrisi-d dataset @xcite , which contained 2,170 sentences. the validation and test sets were composed of 333 and 791 sentences, respectively. within the training dataset, there were a total of 3,893 location mentions, with 763 unique mentions. ## system overview llms have proven to excel in a wide range of linguistic tasks, due to their extensive training on vast amounts of internet data. with their high capabilities in tasks such as generation and reasoning. llms have extensively been studied and utilized in languages like english. however, their potential in the arabic language has not been fully explored, presenting a significant opportunity to investigate and harness llms for arabic nlu tasks. to seize this opportunity, we conduct experiments on several large language models that have been trained on data of multiple languages including arabic, namely llama3, wizardlm-2 @xcite , acegpt @xcite , and openchat @xcite , to assess their performance in word sense disambiguation and location mention disambiguation tasks. in addition, we compared their performance with pre-trained language models (plm) such as marbert @xcite and @xcite . in the following section, we provide detailed information about their training and inference setup. ## results and discussion this section presents the model's performance during the development and testing phases using the official metrics. additionally, an error analysis is carried out to pinpoint the shortcomings of the presented models. for subtask 1 the official metric is the macro average f1-score, while for subtask 2 the official metric is the mean reciprocal rank (mrr@k), where k indicates the cutoff of the retrieved ranked list of candidate toponyms. for devphase results refer to appendix c. ## conclusion this paper presented the work conducted during the arabicnlu shared task. the best solution for both subtasks was based on llama3, which demonstrated superior performance compared to marbert and arabert. however, it is important to note a limitation observed during the experiments, namely the issue of hallucination, where the llama3 model occasionally provided incorrect output formats, or would continue text rather than solving task. future work will focus on enhancing these models to address this limitation and further improve their accuracy and reliability