DRU at W ojood NER 2024: ICL LLM for A rabic NER (2024)

Abstract: This paper details our submission to the WojoodNER Shared Task 2024, leveraging in-context learning with large language models for Arabic Named Entity Recognition. We utilized the Command R model, to perform fine-grained NER on the Wojood-Fine corpus. Our primary approach achieved an F1 score of 0.737 and a recall of 0.756. Post-processing the generated predictions to correct format inconsistencies resulted in an increased recall of 0.759, and a similar F1 score of 0.735. A multi-level prompting method and aggregation of outputs resulted in a lower F1 score of 0.637. Our results demonstrate the potential of ICL for Arabic NER while highlighting challenges related to LLM output consistency.

## introduction named entity recognition (ner) is an essential component of information extraction that involves identifying and classifying named entities in a given text into predefined categories such as names of organizations, people, and locations. ner acts as an important pre-processing step for a variety of downstream tasks including text summarization and question answering @xcite . figure 1 shows an arabic ner example. flat ner identifies non-overlapping entities in a text, while nested ner recognizes entities within other entities such as 'strip' in the third row which is a geopolitical entity nested within the event entity. datasets and methodologies differ for each ner type, with specific corpora and annotation techniques for each one. the wojood corpus @xcite is a significant contribution for arabic ner, containing about 550k tokens in modern standard arabic and various dialects, annotated with 21 entity types such as person, organization, and location. the wojood-fine corpus @xcite extends this with 31 subtypes for categories such as geopolitical entities and facilities. both corpora include flat and nested versions. the wojoodner shared task series @xcite aims to advance arabic ner research using wojood and wojood-fine. the 2024 shared task @xcite introduced new subtasks, including an "open track" subtask (subtask 3) encouraging innovative systems using external datasets and tools, focusing on ner's real-world utility through generative models. this paper presents our submission to the open track for the flat ner subtask (subtask 3a). in-context learning (icl) with large language models (llms) has emerged as a promising approach for ner. icl enables llms to perform tasks by utilizing prompts with a few demonstrative examples per category. our method proceeds as follows: ## related work recent studies explored different techniques and frameworks to enhance icl for ner. gpt-ner transforms sequence labeling into a generation task, classifying entities with corresponding tokens. to mitigate hallucination, self-verification prompts the llm to confirm if the extracted entities belong to a label @xcite . c-icl enhances entity extraction by using correct and incorrect samples in prompts, improving the model's accuracy through reasoning and error correction @xcite . similarly, p-icl uses point entities as auxiliary information in prompts, enhancing the llm's ability to recognize each entity type @xcite . for few-shot nested ner, researchers developed a framework that includes a prompt with task instructions, demonstrations, and possible labels, and developed an example selection mechanism called ende retriever to optimize prompt effectiveness @xcite . researchers evaluated gpt-3.5 and gpt-4 for arabic nlp tasks @xcite , including ner, using zero-shot learning and further evaluated gpt-4 using few-shot learning on datasets such as anercorp @xcite , aqmar @xcite , and qasr @xcite . the study revealed patterns of errors for sequence tagging tasks like ner and pos tagging that included format deviations and incorrect token generation which led to performance drops. ## dataset the ner shared task dataset for subtask 3 includes data from five news domains related to the war on gaza. the subtask provides training and development sets that were manually annotated with fine-grain named entities using the annotation guidelines described in @xcite . the dataset includes both flat and nested versions. the training set contains 1,928 sentences with a total of 50,144 tokens, while the development set has 372 sentences with 10,049 tokens, as shown in ## llm selection after evaluating several open source llms, we selected cohere's command r model @xcite as our chosen llm. command r is an open-weights generative model specifically optimized for long-context tasks such as retrievalaugmented generation (rag). this model contains 35 billion parameters and supports a context length of up to 128,000 tokens, making it well-suited for extensive text processing tasks. command r is proficient in handling 10 major languages: english, french, spanish, italian, german, portuguese, japanese, korean, arabic, and chinese. notably, it ranks as the second-best model on the open arabic llm leaderboard @xcite , which assesses the performance of arabic llms. the evaluation datasets include native arabic benchmarks such as alghafa (almazrouei et al., 2023) and acva @xcite , focusing on reasoning, language understanding, and commonsense tasks. despite its smaller parameter size, command r achieved an average score of 54.43, closely trailing the llama3 model with 70 billion parameters, which scored 59.86. this performance underscores command r's efficiency and capability. for our experiments, we employed the 8-bit quantized version of the command r model, which balances performance and resource utilization effectively. ## prompt design to explain the task to the model, we devised a detailed system prompt. it outlines the steps for tagging tokens according to the wojood-fine annotation guidelines @xcite . the prompt helps the model predict different tag levels for each token and specifies the required output format, ensuring correct parsing for evaluation. the full prompt is listed in appendix a. the initial sentences in the prompt instruct the llm to perform highly accurate ner for arabic text. the model is expected to understand arabic thoroughly and predict the appropriate named entity tags for each token in a provided list, following structured steps. to tag the tokens correctly, the model needs to predict up to three levels of tags. the first level is the high-level tag from wojood's original 21 tags. the second level is a subtype, applicable only for gpe, org, fac, and loc tags. the third level is needed only if the second level tag is gpe_org or org_fac, identifying the specific subtypes from gpe and org, or org and fac. to simplify the problem, the prompt instructs the model to predict only two tag levels: l1 (highlevel tag) and l2 (subtype). it lists possible l2 tags based on l1 (gpe, org, fac, or loc) and their definitions. the model is instructed to output predictions in csv format: token in the first column, l1 tags in the second, and l2 tags in the third, with the most probable tags listed first. to handle cases where gpe_org or org_fac are subtypes, these tags are removed from the l2 list. instead, the model is instructed that l1 tags can include both gpe and org, or org and fac. few-shot examples demonstrate this, e.g., l1 could be "b-gpe b-org" and l2 "b-country b-gov," or l1 "b-org b-fac" and l2 "b-edu b-building-or-grounds." we reconstruct the correct subtype tag in post-processing. the prompt also instructs the model to follow the bio schema, with "b-" marking the beginning and "i-" marking the inside of an entity. we added 27 illustrative examples of input sentences and output predictions to the prompt. these sentences were chosen manually, preferring shorter sentences with diverse examples and more non-"o" tags. the format of the few-shot examples is shown in appendix b. ## output quality issues after inspecting command r's generated outputs, we identified several quality issues. firstly, the model sometimes produced extra tokens or missed tokens for a given input sentence. although there are 10,049 tokens in the development set, the model outputted only 9,915 tokens because some predicted sentence outputs were either shorter or longer than their corresponding ground truth sentences. secondly, the model tagged some tokens with tags it had not encountered in the prompt before, resulting in 14 different hallucinated tags. thirdly, only 9,416 out of the 9,915 token predictions generated by the model strictly conformed to the specified output format. for the remaining tokens, there were several instances of incorrect output formats. we observed a total of six distinct cases of wrong output format that are amenable to postprocessing as shown in appendix c. although the formats were syntactically incorrect, their tags semantically made sense upon inspection. we handled the six cases using a post processing step and we report results with and without post-processing. ## split multi-level prompt method we also experimented with smaller prompts targeting subsets of named entity types and aggregated the results. first, we created separate prompts, prompt1 (11 wojood ner enitities) and prompt2 (remaining 10). they instruct the model to tag only using the provided subset. for the output of the first subset, we collected the tokens classified with tags that have subtypes (gpe, org, fac, and loc) and asked the llm to classify the extracted entities into the corresponding subtypes. we created four different icl prompts for that. we reconstructed the predicted tags from both levels. if a sub-tag was gpe_org or org_fac, we report the third-level tags from the results of the corresponding prompts (gpe and org prompts or from org and fac). ## discussion we discuss several key insights and challenges using command r model for arabic ner. the 0.737 f1 and 0.756 recall primary scores indicate an effective performance. post-processing resulted in slight recall improvement with a similar f1 score. this suggests that while format corrections helped recover some missed entities, it also impacted the precision. adherence to output formats may be better with finetuning approaches to avoid unnecessary performance penalties. the multi-level method resulted in a lower f1 score of 0.637. we hypothesized that shorter, more specific prompts help the model focus better. however, separating entities into distinct prompts might have disrupted the llm contextual understanding. it seems llms work better when performing multiple tasks that feed into each others. the potential for error propagation in aggregating multiple prompts may have introduced errors as well. ## conclusion in this paper, we explored using llms for arabic ner through icl with the command r model. our experiments indicate that while the model has potential for arabic ner tasks when provided with high-quality prompts, it occasionally deviates from the specified output format, necessitating additional post-processing. future work could focus on better example retrieval for prompts, as the selection of the fewshot examples could significantly impact in-context learning performance @xcite . additionally, developing more robust prompting strategies to enforce format consistency and comparing different llms to understand their strengths and limitations in arabic ner could yield valuable insights. these improvements could enhance the efficiency and scalability of llms for ner tasks.