El-Kawaref at W ojood NER shared task: S taged NER for A rabic Named Entity Recognition (2023)

Abstract: Named Entity Recognition (NER) is the task of identifying word-units that correspond to mentions as location, organization, person, or currency. In this shared task we tackle flat-entity classification for Arabic, where for each word-unit a single entity should be identified. To resolve the classification problem we propose StagedNER a novel technique to fine-tuning NER downstream tasks that divides the learning process of a transformer-model into two phases, where a model is tasked to learn sequence tags and then entity tags rather than learn both together simultaneously for an input sequence. We create an ensemble of two base models using this method that yield a score of on the development set and an F1 performance of 90.03% on the validation set and 91.95% on the test set.

## introduction named entity recognition (ner) is a vital sub-task for a plethora of nlp applications, those of which include machine translation @xcite , co-reference resolution @xcite and information extraction @xcite . the sub-task exhibits challenges when addressed from the lens of arabic data, this comes back to the fact that the language is one of the richest in morphological inflections. to add more, attributes that typically help in locating entities such as capitalisation is not featured in the language. arabic is also agglutinative in nature where one word could be combination of lemma, prefixes and suffixes @xcite ) @xcite . arabic ner (aner) has been approached using a wide spectrum of methods through the years however, more recently development of pre-trained language models (plms) specifically transformer-based models that learn context-aware representations has elevated the performance on aner datasets. these models include marbert and arbert @xcite , @xcite . the architecture of these plms has been extended and equipped with different networks. to exemplify, (al-qurishi and souissi, 2021) utilized a range of transformer based models namely arabert, xlm-roberta @xcite and @xcite coupled with conditional random field (crf) to fine-tune an aner downstream task revealing that arabert exhibited the highest scores. bilstm and bigru-crf models have also been fine-tuned on arabic bert in an attempt to classify entities based on classical arabic. @xcite . in similar vein, we leverage transformer based models to classify flat entities. however, we employ an alternative technique to fine-tuning plms on ner tasks where the learning regiment for a model is distributed over two stages for better learning @xcite . in the next sections we begin by describing the data purposed for this shared task @xcite in section 2 and highlight how we re-purposed train, validation and test sets to perform a two-staged fine-tuning process. next, we give an extensive explanation of our adopted fine-tuning method in section 3. in sections 4 and 5 we present results of the submitted system, and discuss and analyse system performance on the validation set. finally, we summarize and recap the proposed system and re-highlight performance scores and findings in section 6. the corpus is also further extended to include annotation for nested entities, however for the scope of this shared task paper only annotations purposed for flat entities are used. the total number of tokens amounts to over 550k of modern standard arabic (msa) and dialectical arabic tokens. to add more, msa tokens are more frequent, where about 86% of tokens are msa and the rest come in the levant dialect. the corpus covers a different domains for each arabic class; msa tokens were acquired from two resources, the birzeit university digital palestinian archive, "awraq", and online articles 1 . the former covers cultural heritage and modern history of palestine while the latter includes web articles of health, law, finance, politics, migration, terrorism, ict and elections. meanwhile, dialectical tokens were obtained from supplementary lebanese and palestinian corpora @xcite @xcite and other additional levant resources collectively discussing general topics. train, development and test splits were provided; as depicted in figure 1 out of word vocabulary (o) instances exceed other entity classes, where there were about 258k and 36k in the train and validation sets of o instances, for that reason figure 1 brings focus to other less dominant meaningful entities; whereby date, organization, geopolitical, occupation and person are recurring throughout the data in comparison to language, product, quantity, currency and unit which are rarely present. as briefly mentioned before, the core idea of the proposed learning technique relies on separation of learning of sequence labels (bio) tags and entity classes, hence the data goes through a separation of sequence labels and entities. moreover, we rely on arabert's pre-processor foot_1 whereby diacritics and elongations are removed by default. ## system description the backbone of the submitted system relies on fine-tuning a language model based on bert's transformer architecture @xcite . typically, data utilised in the fine-tuning process for ner tasks follows bio format, whereby at word-level, each entity is accompanied by an appropriate b (beginning) or i (inside), or o (outside) tag, hence the model is tasked to learn a position of an entity and the entity itself altogether. meanwhile, we adopt the stagedner approach @xcite whereby the learning process is split into two sub-tasks, the first mimics a sequence-labelling problem where the model learns to assign appropriate bio tags for each input, and the second sub-task is the original entity classification task. we note that this method is not analogous to sequential learning, as two separate instances of a transformer are leveraged in this method, thus each instance is assigned to exclusively learn either a bio tag or an entity class. ## results we report micro f1, precision and recall scores for development and test sets in tables 4 and 5 for every ensemble and their unique pos combination. additionally we showcase our performance compared to other teams in by inspecting ## conclusion in this shared task, we tackled flat entity classification on wojood corpus, a predominantly msa dataset, where we applied an alternative fine-tuning method, where one model is used to learn bio tags and another separate model is used to learn entity classes, instead of a single model that learns to perform both tasks jointly. the motivation behind this was to lessen the number of classes a model had to learn; where instead of learning sequence variations of one entity such as i-org, b-org, the model simply learns to identify org and another model is tasked to learn bio sequence tags. to strengthen the learning of bio tags we equip the model with msa and levant pos tags and created four ensembles based on different combinations of them. results show that having msa pos tags made a difference in performance where the best performing ensemble that include msa pos tags scored 90.03% and 91.95% on the development and test sets respectively. our best performing model can be demonstrated on huggingface spaces 4 .