muc AI at W ojood NER 2024: A rabic Named Entity Recognition with Nearest Neighbor Search (2024)

Abstract: Named Entity Recognition (NER) is a task in Natural Language Processing (NLP) that aims to identify and classify entities in text into predefined categories.However, when applied to Arabic data, NER encounters unique challenges stemming from the language’s rich morphological inflections, absence of capitalization cues, and spelling variants, where a single word can comprise multiple morphemes.In this paper, we introduce Arabic KNN-NER, our submission to the Wojood NER Shared Task 2024 (ArabicNLP 2024). We have participated in the shared sub-task 1 Flat NER. In this shared sub-task, we tackle fine-grained flat-entity recognition for Arabic text, where we identify a single main entity and possibly zero or multiple sub-entities for each word.Arabic KNN-NER augments the probability distribution of a fine-tuned model with another label probability distribution derived from performing a KNN search over the cached training data. Our submission achieved 91% on the test set on the WojoodFine dataset, placing Arabic KNN-NER on top of the leaderboard for the shared task.

## introduction named entity recognition (ner) is a downstream task within natural language processing (nlp) that entails identifying the entity type for each word in a given sentence. these entities usually belong to predefined tags such as persons, locations, or dates. ner has proven beneficial for several downstream tasks, including relation extraction, machine translation, co-reference resolution, and information extraction. the significance of the ner task has led to the development of various approaches, including span-based classification @xcite , sequence labeling, and sequence-to-sequence generation. recently, with the widespread adoption of large-scale pre-trained language models (plms), in-context learning-based approaches @xcite have also emerged as a prominent method. however, applying ner to arabic data presents additional challenges. unlike english, arabic does not use capital letters at the beginning of words, making it more challenging to identify nouns and determine the start and end of entities. multiple word variants can also refer to the same semantic meaning @xcite . furthermore, arabic is rich in morphological inflections, meaning that a single word can consist of multiple morphemes @xcite . finally, annotated arabic corpora for the ner task are limited compared to those available for english @xcite . previous attempts to enrich the arabic ner corpus as the multilingual dataset ace @xcite , anercorp @xcite , and ontonotes5 @xcite . more recently, the wojood @xcite dataset was introduced, a large-scale arabic ner dataset collected from multiple sources covering modern standard arabic (msa) and dialects. however, all of these datasets are all annotated with coarse-grained entity types @xcite . the most recent corpus is wojoodfine @xcite , which extends wojood by providing 31 fine-grained annotations, introducing subtypes for some of the main entity types. in this paper, we tackle the shared subtask flat ner with subtypes using the wojoodfine dataset; we comprehensively describe the data in section 3. we propose enhancing a fine-tuned ner model by integrating knn search over the training entities during the inference phase. knn-ner @xcite is a framework that can be applied to models that have already been fine-tuned, requiring no additional training or fine-tuning. applying the knn-ner framework has achieved a 91% micro-f1 score and placed us on top of the leaderboard for the shared subtask 1 flat ner. ## task definition the shared subtask 1 is flat ner with subtypes. in this task, for a given sentence, the objective is to identify and classify named entities, which may span multiple words. for each identified entity, the task is to determine the main entity type and up to two levels of subtypes, if they exist. for example, in figure 1 (b) , the main entity date spans two words and has no subtypes. meanwhile, the entity org spans four words and has two levels of subtypes. the first-level subtype is org_fac (a subtype of org), and the secondlevel subtypes are com (a subtype of org) and building-or-ground (a subtype of fac). more formally, given a sequence of tokens t of length m denoted as @xmath0, t 2 , ...., t m ), the goal is to identify and output a set of named entities. each named entity is represented as a tuple (s, e, main_tag, [sub_tags]), where s and e are the start and end tokens of the entity, respectively. the tuple also includes the main entity tag and up to two levels of subtypes. ## data we conducted our work in wojoodfine dataset @xcite provided in the shared task @xcite . wojoodfine is an extended version of wojood @xcite dataset, which is a ner dataset with 550,000 tokens manually annotated across 21 entity types; approximately 80% of wojood's data was sourced from modern standard arabic (msa) articles, while 12% was gathered from social media content in palestinian and lebanese dialects. wojoodfine extends the wojood dataset by providing fine-grain annotations for entity sub-types. namely, each of the words that has one of the main entity types (geopolitical entity (gpe), organization (org), location (loc), and facility (fac)) can have from zero to multiple subtypes from the predefined 31 subtypes. for example, in figure 1 (a) , words with the same main entity can have different subtypes. the train and development entities distribution can be found in figure 2 . for the exact mapping of main entities to subtypes, we refer the reader to check @xcite . ## system description our approach is centered on fine-tuning a language model based on bert's transformer architecture @xcite . our methodology is inspired by @xcite , which employs a two-step process: joint vanilla fine-tuning followed by knn at inference time. ## discussion the results from our experiments demonstrate the effectiveness of incorporating knn search at inference time for the flat ner task in arabic. the comparison between vanilla fine-tuning without and with knn search reveals a consistent improvement in the f1-score, as shown in we show in figure 4 the sensitivity of our system's micro f1-score for different values of n and interpolation factor λ. if the interpolation factor is one (λ = 1), the final distribution converges to only the baseline, using only knn-rr appears to be competitive or even better than using the fine-tuned model for small values of n. however, its performance drops to 88.5% when @xmath1. this decline can be attributed to class imbalance between entity tags and the label o. as the number of retrieved neighbors increases, more neighbors with the label o are retrieved, thereby increasing the probability mass of the label o in p kn n . ## limitations and future work a limitation of this study is that knn-ner was not evaluated on multiple models, which limits the assessment of its robustness and model-agnostic property of the knn-ner framework. another limitation is the increased inference time. since knn requires searching for labels in the datastore during inference, the overall inference time is extended by both the model's processing time and the additional time required for the knn search. this increased inference time may affect the practicality and efficiency of our model in real-world applications. another limitation is that we did not explore other similarity scores than cosine similarity, nor assessed the quality of the similarity scores it produced. finally, future work should include exploring applying knn-ner to subtypes. ## conclusion in this shared task, we tackled flat ner with subtypes on the wojoodfine corpus, where we trained the model jointly with two mlp heads, one for predicting the main entity and the other for predicting possibly multiple subtypes. we finetuned arabert and applied knn search over the training set during inference time to enhance the model's capability of predicting the main entity type for each token in the test set. the motivation behind incorporating knn search was to improve the model's performance without requiring any further training after the initial fine-tuning phase. this approach aimed to efficiently utilize the trained model by leveraging knn over the training set. the results show an improvement with the incorporation of knn search. specifically, the micro-f1 scores for the development set increased from 91.62 to 92.90, and for the test set from 90.09 to 91.00, indicating a robust enhancement in performance. our approach ranked first in the shared task leaderboard.