DRU at W ojood NER 2024: A Multi-level Method Approach (2024)

Abstract: In this paper, we present our submission for the WojoodNER 2024 Shared Tasks addressing flat and nested sub-tasks (1, 2). We experiment with three different approaches. We train (i) an Arabic fine-tuned version of BLOOMZ-7b-mt, GEMMA-7b, and AraBERTv2 on multi-label token classifications task; (ii) two AraBERTv2 models, on main types and sub-types respectively; and (iii) one model for main types and four for the four sub-types. Based on the Wojood NER 2024 test set results, the three fine-tuned models performed similarly with AraBERTv2 favored (F1: Flat=.8780 Nested=.9040). The five model approach performed slightly better (F1: Flat=.8782 Nested=.9043).

## introduction named entity recognition (ner) is a crucial task in natural language processing for detecting and categorizing named entities like personal names, locations, and organizations in text. it has diverse applications, such as extracting genes and diseases in biomedical texts @xcite , identifying financial entities in finance and legal documents @xcite , and recognizing names and locations in social media posts @xcite . named entities are classified as flat or nested. flat entities are contiguous words with no overlap, while nested entities have a layered structure where one entity can contain another. this complexity has led to the development of specialized tools and models for both types. in the development of arabic ner, the wojood corpus @xcite was devised which provides a robust foundation for arabic flat and nested ner. it includes 550k manually annotated tokens across 21 entity types. the wojood corpus was further transformed into wojood-fine @xcite , which introduces 31 new fine-grained sub-types for four key entity categories: geopolitical entity (gpe), location (loc), organization (org), and facility (fac). to further accelerate arabic ner research, the wojoodner shared task series @xcite was introduced. the 2024 shared task @xcite utilizes the wojood-fine corpus through two sub-tasks: both sub-tasks are confined to the use of the given wojood-fine training and development sets. this paper presents our submissions to the wojoodner 2024 shared task and offers the following contributions: the um6p and ul team developed a bertbased multi-task learning model for flat and nested arabic ner. they used arabic pretrained language models (plms) to encode sentences and combined several training objectives to boost performance. among the plms evaluated were, qarib @xcite , camelbert-mix @xcite , and arbertv2 which yielded the best results (el @xcite . the alexu-aic team used sequence labeling and machine reading comprehension (mrc) techniques for arabic ner. they applied an mrcbased approach for flat ner and fine-tuned a plm for nested ner. using the jaber @xcite pre-trained model for sequence labeling, they created queries for each entity category in the mrc method and enhanced results with stochastic weighted averaging (swa) @xcite . the lipn team used a span-based approach with a plm for token representation and neural network classifiers for both flat and nested arabic ner. they applied global decoding for flat ner and a greedy strategy for nested ner, involving token encoding, span enumeration, and classification @xcite . ## data we provide here a description of the released datasets by the shared task organizers. the wo-joodfine corpus is offered in two versions, flat, and nested. training, development, and test splits are available in both versions. ## multi-label token classification we applied the same training process for all our model iterations. the only change was in input classification. we one-hot encoded b-(beginning) and i-(inside) type and sub-type token tags. we utilized binary cross-entropy with logits loss as the primary loss function, handling multiple overlap-ping entity types per token. this function is particularly useful for binary classification. literature suggests this works with multi label ner @xcite . the function combines a sigmoid and binary cross-entropy under one layer. it is more numerically stable than using a plain sigmoid followed by a binary cross-entropy loss due to expecting logits exposed to inputs. therefore, in order to obtain the output probabilities for each class, we directly obtain output probabilities and subject them to thresholds. to accommodate for variations, we apply a pre-processing step before training for classification, and ignore tokens pertinent to specific model architectures. we use -100 label across the one-hot encoded vector, and ignore them in prediction. we trained all iterations on the nested dataset, and extracted the highest level entity type and corresponding sub-types to infer the flat prediction. ## type/sub-type classification method we first attempted to train different models, with different architectures, on predicting all types and sub-types in one shot. the given model would take as input a 103 length one-hot encoded vector, for each of the b-and i-types and sub-types, and the missing tag notation "o". we experimented with bloom, gemma, and bert. bloomz-7b-mt @xcite ) model has 7 billion parameters. it augments bloomz with multilingual mt0 capabilities @xcite . we further fine-tuned this model on a high-quality arabic dataset, including arabic high school curricula books, and specialized books in social sciences and humanities. our second option, gemma, is a 7 billion parameter model, pre-trained on a wide variety of languages including arabic. arabertv2 is the third and lightweight option. it follows bert architecture and is trained on arabic wikipedia, and quality newspapers such as assafeer. ## one×1 classifiers method we separated type and sub-type classification and dedicated a model for each. in our first iteration of this design, we trained one instance of arabertv2 to exclusively predict main types. we trained another instance to predict sub-types. we then eval-uated the data across both models and aggregated their outputs, compensating for missing tags and rooting to improve recall. ## one×4 classifier method instead of only one model for sub-types, we trained four instances, each specialized in the sub-types of a specific group: gpe, org, fac, loc. the other main types have no subtypes. experiment 1 predicts the 21 main types with no changes. the four specialized encompass all the sub-types. experiment 2 allows predicting gpe with org and org with fac. at the subtype level, the gpespecialized model does not predict gpe_org and the org-specialized model does not predict org_fac. the intuition is that this reduces class intersection confusion for the main types model. for example, if an entity was initially tagged as gpe and had sub-entity tags: gpe_org, coun-try, and gov, we transformed gpe_org to org. the final output aggregates the results from both stages. ## system description larger models, namely bloomz and gemma. were fine-tuned with a sequence length configuration of 512 tokens, for 12 epochs on a batch size of 8 for training and evaluation. model fine-tuning was executed on 1x nvidia a100 80 gb vram gpu. bert-based models were trained for 10 epochs on a batch size of 16, on 1x nvidia a4000 16 gb vram gpu. ## results we experimented with bloomz, gemma, and arabertv2. all models performed similarly, with arabertv2 favored slightly (f1 0.8780 and 0.9040 for flat and nested respectively). we experimented with the one×1 setup, where one stage was dedicated to predicting main types and another for sub-types. our first iteration performed adequately, but was still outscored by a single-layer approach in both tasks (0.8780 and 0.9040 compared to 0.8746 and 0.9033 in f1). one×4 experiment 2, where we resolved subclass interactions, proved to be the most effective, featuring the best performance on both tasks with 0.8782 and 0.9043. based on the development set provided, we calculated evaluation metrics per entity. the results presented in ## conclusion & future work this study explored three attempts to enhance the performance of wojoodfine: fine-tuning with different architectures, specializing a model for predicting types and another for sub-types, and further differentiation by developing individual models for each sub-type category. among these strategies, training a model for the main tag and models on each sub-type category scored the highest accuracy on sub-tasks 1 and 2 with 0.8782 and 0.9043 f1 scores respectively.