mu NER a at W ojood NER 2024: Multi-tasking NER Approach (2024)

Abstract: This paper presents our system “muNERa”, submitted to the WojoodNER 2024 shared task at the second ArabicNLP conference. We participated in two subtasks, the flat and nested fine-grained NER sub-tasks (1 and 2). muNERa achieved first place in the nested NER sub-task and second place in the flat NER sub-task. The system is based on the TANL framework (CITATION),by using a sequence-to-sequence structured language translation approach to model both tasks. We utilize the pre-trained AraT5v2-base model as the base model for the TANL framework. The best-performing muNERa model achieves 91.07% and 90.26% for the F-1 scores on the test sets for the nested and flat subtasks, respectively.

## introduction the abundance of written texts in the digital age has increased the importance of identifying key pieces of information, known as named entities, within the text. this process, called named entity recognition (ner), involves identifying and classifying entities into pre-defined categories, such as names of persons, organizations, locations, dates, and more, depending on the application. the ability to accurately identify these entities is crucial for numerous real-time applications, including biomedical and clinical studies @xcite , business @xcite , and law @xcite . moreover, ner complements other nlp tasks, including question answering (guven and unalir, 2021), machine translation @xcite , and coreference resolution @xcite . this paper presents our system, munera, which participated in the wojoodner 2024 shared task at the second arabicnlp conference @xcite . the wojoodner shared task, now in its second year, includes two subtasks: flat and nested fine-grained ner. also, there is a third "open track" subtask in which we did not participate. wojood-ner aims to advance the state-of-the-art in arabic ner by providing a large-scale, fine-grained, and annotated dataset and a competitive platform for researchers to evaluate their models. the main difference between wojoodner 2024 and 2023 is the introduction of wojood fine with finer-grained sub-tags and sub-sub-tags @xcite . also, there has been a major revision of the tagging schemes. typically, ner is modeled as a sequence classification problem such that each token is assigned a tag using a sequence tagging scheme such as iob2 @xcite . in munera, we model both the nested and flat ner tasks as a sequence-to-sequence translation utilizing the translation between augmented natural languages (tanl) framework @xcite . notably, all participating teams in the wojood 2023 shared task @xcite did not utilize sequence-to-sequence (encoder-decoder) architecture, their work employed bert pre-trained language models (encoder-only) in different ways @xcite @xcite @xcite . ## work on wojoodner 2023 wojoodner-2023, the first arabic ner shared task @xcite , includes flatner and nestedner subtasks with arabertv2 as the baseline @xcite . notably, most participating teams utilized bert pre-trained language models, with the performance of the encoderdecoder t5 model remaining unexplored @xcite . seven studies employed @xcite , and it is remarkable that arabert consistently outperformed other transformers: lipn @xcite achieved first place in flatner, elyadata @xcite excelled in nestedner. el-kawaref @xcite utilized stagedner, while um6p & ul (el mahdaouy et al., 2023) adopted multi-task learning. additionally, lotus @xcite explored multi-task learning with xlm-r, arabert, and marbert, favoring xlm-r. alexu-aic @xcite employed a machine reading comprehension-based approach, leveraging various plms, while alex-u 2023 nlp @xcite introduced ara-binder, utilizing bert models. in our work, we explore a new method utilizing the t5 encoder-decoder architecture instead of the bert encoder architecture. ## data the wojood corpus is a dataset for arabic ner dataset containing approximately 550,000 tokens of modern standard arabic and dialects. this dataset is designed specifically to identify nested entities, where entities are embedded within other entities. the wojood corpus consists of two versions for arabic nested ner: the original and the wojood fine corpus @xcite . ## methodology we utilize the tanl framework @xcite to train our models for both flat and nested ner. the tanl framework is designed to address structured prediction tasks in language processing. these tasks include joint entity and relation extraction, relation classification, semantic role labeling, event extraction, coreference resolution, dialogue state tracking, and nested named entity recognition @xcite . one key advantage of the tanl framework is its ability to incorporate the semantic understanding of labels during model training. unlike traditional task-specific classifiers, which typically train without explicit knowledge of label semantics, tanl leverages pre-trained models' understanding of entity semantics (e.g., person, location). in the tanl framework, depending on the task, both the input and output are structured in augmented natural languages. in our case, we created the ner structure task in tanl similar to the adverse drug effect (ade) dataset @xcite structure task, as both datasets feature nested ner entities. the task aims to extract entities and their spans from a given sentence. these entities are enclosed within the special tokens [ ], and each entity is followed by its type and separated by |, e.g., [ token | entity type ]. entity types are represented in natural language words-such as person or location-rather than abbreviations like per or loc to leverage the model's semantic knowledge of these words. for wojoodner labels, we manually created a dictionary mapping the entity tags (e.g., per) to their natural language words (e.g., person) based on the description of the tags in the wojood fine paper @xcite . some entity tags are already in a natural language format, such as "product" and "sport", while others require extracting the appropriate natural language terms from their descriptions as shown in after generating the output, misspellings of the tokens in the original text or minor format errors can occur due to the nature of generative models. to address this issue, tanl's decoding process first cleans the output by removing special tokens and discarding invalid formats, ensuring only valid entity types are considered. to further enhance robustness, tanl utilizes the dynamic programming (dp) based needleman-wunsch alignment algorithm @xcite , which aligns input and cleaned output tokens at the token level. for instance, it can correctly align a misspelled generated token (e.g., ) with its correct form in the input (e.g., ), ensuring accurate entity identification. we noticed that most of the invalid outputs are: ## experiments and results we use the tanl model @xcite , based on arat5 v2 @xcite as the primary classifier, and fasttext (ft) @xcite trained on spans from the training set as a secondary classifier. for ft, we use the pre-arabic trained vectors fasttext-ar-vectors @xcite , which is available on huggingface foot_0 . for tanl, we set the values for hyperparameters as follows: hyperparameter value learning rate (lr) 5e -4 beam size 8 number of epochs 5 train batch size 4 max sequence length 768 to train the tanl model, we use microsoft azure ml service with the nc a100 v4 series, powered by nvidia a100 gpus with 80 gb memory. for inference, we use google colab pro+ with one nvidia a100 gpu. the results indicate that the best model on both tasks (i.e., nested and flat) is tanls, which tags all levels without additional tagging using the secondary classifier ft. we observe that all tanlbased models perform well in span detection and recognition. additionally, when using fasttext as a secondary tagger with tanlp, the results indicate that the performance is close to tanls for nested (less than 1% difference in f-1), but not the case with the flat subtask. this may be due to overpredicting entity types by fasttext, as reflected in the precision and recall scores. for tanl with the simpler tagging scheme (i.e., tanlp), the model performs better than the model with subtags in span detection for the flat subtask. however, for the nested subtask, the model trained on subtags performs better than the model trained on only parent tags, tanlp. ## conclusions this paper presents the results of "munera" on the wojoodner 2024 shared task. the munera system demonstrated outstanding performance in the wojoodner 2024 shared task, securing first place in the nested ner sub-task and second place in the flat ner sub-task. the system's success is primarily attributed to the innovative application of the tanl framework @xcite , combined with the arat5v2-base model @xcite . these methodologies allowed munera to achieve remarkable f-1 scores of 91.07% for nested ner and 90.26% for flat ner. our system shows that the encoder-decoder architecture tackles the complexity of the wojood fine corpus annotation scheme, effectively modeling the multi-layered tagging scheme and the challenge of the nested entities structure. despite the challenges posed by inconsistencies and sparse examples in the dataset, munera consistently delivered high-precision and recall metrics. for future work, there are several directions. one is integrating the hierarchical structure of the 3-level tagging scheme (i.e., tags, sub-tags, and sub-sub-tags) into the model, potentially improving performance. an example is experimenting with different structures for the tanl format that incorporate the hierarchical tagging scheme. additionally, employing more advanced secondary taggers, like bert, could improve the performance further than our secondary tagger, fasttext. finally, addressing the computational limitations and experimenting with a broader range of hyperparameters will also be crucial for future advancements.