Upaya at A rabic NLU Shared-Task: A rabic Lexical Disambiguation using Large Language Models (2024)

Abstract: Disambiguating a word’s intended meaning(sense) in a given context is important in Nat-ural Language Understanding (NLU). WSDaims to determine the correct sense of ambigu-ous words in context. At the same time, LMD(a WSD variation) focuses on disambiguatinglocation mention. Both tasks are vital in Nat-ural Language Processing (NLP) and informa-tion retrieval, as they help correctly interpretand extract information from text. Arabic ver-sion is further challenging because of its mor-phological richness, encompassing a complexinterplay of roots, stems, and affixes. This pa-per describes our solutions to both tasks, em-ploying Llama3 and Cohere-based models un-der Zero-Shot Learning and Re-Ranking, re-spectively. Both the shared tasks were partof the second Arabic Natural Language Pro-cessing Conference co-located with ACL 2024.Overall, we achieved 1st rank in the WSD task(accuracy 78%) and 2nd rank in the LMD task(MRR@1 0.59)

## introduction the first shared task is word sense disambiguation in arabic @xcite , which aims to determine a word's intended meaning (sense) in a given context. word sense disambiguation (wsd) is a long-standing challenge in natural language processing (nlp), and it has an extended history of research. @xcite . in the shared task arabicnlu2024 wsd task @xcite , arabic language adds further challenges in wsd as arabic exhibits morphological richness, encompassing a complex interplay of roots, stems, and affixes, rendering words susceptible to multiple interpretations based on their morphology. second task location mention detection (lmd) is a special case of wsd. disambiguating location/geolocation from social media posts is very useful in disaster management, as it helps response authorities, for example, locating incidents for planning rescue activities and affected people for evacuation. @xcite , an lmd system aims at matching location mentions (lms) appearing in microblogs toponyms, i.e., place or location names, in a geo-positioning database, i.e., gazetteer. @xcite , the lmd problem can be typically decomposed into two sub-problems: 1) candidate retrieval (which aims to retrieve a list of candidate toponyms from gazetteer), 2) and candidate ranking (which aims to rank the list of retrieved candidates). in this paper, we employed zero-shot learning on llama3 (ai@meta, 2024), to solve wsd task as training data wasn't provided. to solve lmd task, we employed openstreetmap api @xcite , for candidate retrieval and we fine-tuned cohere models foot_0 for re-ranking with the training data. ## literature review the task of resolving word sense ambiguity has attracted significant research efforts. these approaches can be broadly categorized into two main paradigms: for arabic specifically @xcite , created a dataset of labeled arabic context-gloss pairs (around 167k pairs) using arabic ontology and benchmarked this dataset using three finetuned arabic bert models. the recent emergence of large language models (llms) such as gpt-3 represents a significant advancement in natural language processing (nlp) @xcite . these models have expertise in a variety of domains, and hence, they can be used as it is in multiple nlp tasks. traditional language models use separate pre-training and fine-tuning pipelines @xcite @xcite , where the fine-tuning stage follows pre-training. models are fine-tuned on a task-specific dataset in a fully-supervised manner. the recent large language models such as llama-3 and gpt-4 @xcite , are improving at handling multiple languages and providing decent accuracy with zero-shot learning / only with prompt instructions (where we define the required task instruction in detail). these models have become the preferred choice when training data is not available. ## wsd with zero-shot learning as there is no training data present in wsd task, we leverage prompt based approaches to solve the task. first, we manually convert the wsd problem statement to natural language based task description and use it consistently for all prompt based experiments. in wsd task, each input consists of arabic sentence, arabic word, and list of definition choices. in our first experiment, we prompt models with wsd task description with input defined above to generate the correct option choice. we refer to this approach as "base prompt". to enforce input format, we convert all parts of an input to a json formatted string and prompt the model to generate output in a fixed schema. we refer to this approach as "structural input and output". we found 6% accuracy improvement for llama-3-70b-instruct model and 0.4% drop in accuracy for gpt-4-turbo. this suggests that gpt-4-turbo's performance is agnostic to input/output format and llama-3-70b-instruct understands structured inputs more accurately than raw inputs. to further handle the complexity of the arabic language, we also explored in-context learning based methodology. specifically, we used llama-3-70b-instruct model to generate arabic sentences for a set of words with their true definitions present in arabic sentences. after getting arabic sentences, we pass these sentences along with true definitions as few-shot examples for the model. we refer to this approach as "structural input and output + in-context learning". however, we found that this approach hurt the accuracy of the dev set. this also suggests that more experiments are required for effective in-context learning capability. ## lmd with llm-based re-ranking the approach consists of two stages: 1. candidate retrieval 2. candidate re-ranking. ## conclusion this work explores the use of large language models for resolving lexical ambiguity in the arabic language. two lexical ambiguities are addressed here: 1. word sense disambiguation (wsd) and 2. location mention disambiguation (lmd). since no training data was available for the wsd task, we employed the llama-3 model with zero-shot learning. our wsd system achieved the 1st rank with an accuracy of 0.7782. for the lmd task, we used cohere models to re-rank the toponym candidates retrieved using openstreetmap api. our lmd system secured the 2nd spot with mrr@1 of 0.599.