Addax at W ojood NER 2024: Attention-Based Dual-Channel Neural Network for A rabic Named Entity Recognition (2024)

Abstract: Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that focuses on extracting entities such as names of people, organizations, locations, and dates from text. Despite significant advancements due to deep learning and transformer architectures like BERT, NER still faces challenges, particularly in low-resource languages like Arabic. This paper presents a BERT-based NER system that utilizes a two-channel parallel hybrid neural network with an attention mechanism specifically designed for the NER Shared Task 2024. In the competition, our approach ranked second by scoring 90.13% in micro-F1 on the test set. The results demonstrate the effectiveness of combining advanced neural network architectures with contextualized word embeddings in improving NER performance for Arabic.

## introduction named entity recognition (ner) is one of the main tasks of natural language processing (nlp) and plays a fundamental role in different nlp applications, such as machine translation, information retrieval, and question-answering @xcite . ner involves extracting entities from unstructured text, including people's names, organizations' names, locations, and dates. these entities can be categorized as flat or nested based on their structure. flat entities consist of a single non-overlapping span of text representing a single distinct entity. nested entities, on the other hand, include entities that are embedded within other entities. in this paper, we present our participating system for the second edition of the flat subtask of ner shared task 2024 @xcite , building on the foundations laid by the first edition (jarrar et al., * these authors contributed equally to this work. 2023). our system is a bert-based model that utilizes a two-channel parallel hybrid neural network with an attention mechanism. the best results were achieved using @xcite . we evaluated our system's performance using the micro-average f1 score, recall, and precision. on the test set, it achieved a micro-f1 score of 90.13% and secured the second position. ## related work previous studies in named entity recognition (ner) have employed diverse methodologies, including knowledge-based approaches, machine learning techniques, and deep learning methods. knowledge-based approaches were among the earliest strategies used in ner. these methods relied on hand-crafted rules @xcite and on language-specific knowledge such as lexical markers @xcite and entity dictionaries @xcite . with the rise of machine learning, studies shifted from hand-crafted rules to various machine learning techniques such as support vector machines (svm) @xcite , hidden markov models (hmm) @xcite , and conditional random fields (crf) @xcite . recently, significant progress in ner has been driven by advancements in deep learning @xcite . various deep learning architectures have been successfully incorporated into ner, including convolutional neural networks (cnns) @xcite , recurrent neural networks (rnns) @xcite , and transformers @xcite . recent models leveraging transformer architectures, such as bert @xcite and its variants, have set new benchmarks in ner performance by capturing more contextual nuances and complex language structures than traditional models. additionally, improvements in pretrained language models @xcite have enabled the adaptation of ner systems developed with fewer annotated data to new languages and domains, improving their accuracy and performance. however, ner still faces many challenges, especially in low-resource languages such as arabic @xcite . most arabic ner research has focused on flat entities, primarily targeting a few coarse-grained entity types, such as person, organization, and location. although coarse-grained ner is useful across various domains and serves as a foundational approach, it falls short for tasks that demand a more detailed understanding of named entities @xcite . in this context, w ojood f ine @xcite was introduced to address the scarcity of fine-grained arabic ner corpora. it is an extension of wojood @xcite that introduced finer-grained sub-types. wojood initially included 21 entity types and was primarily gathered from modern standard arabic (msa) articles, which constituted the majority of the data, with a smaller portion collected from social media in palestinian and lebanese dialects. w ojood f ine enhanced the original wojood corpus by introducing fine-grained entities to four entity types: geopolitical entity (gpe), organization (org), location (loc), and facility (fac). ## data the wojood fine corpus extends annotation of the wojood original corpus @xcite by including fine-grained annotations for named-entity subtypes. it contains 550k tokens and was manually annotated with 21 entity types. it is worth mentioning that approximately 80% of wojood originates from modern standard arabic (msa) articles, but about 10% comes from content on social media sites that were written in palestinian or lebanese dialects via the curras and baladi corpora @xcite . additionally, wojood has also nested named entities, though some of them could refer to more than one type of entity like 'organization', which may interfere with information retrieval tasks further downstream. in response, w ojood f ine provides subtypes for four main categories of entities: gpe, org, loc, and fac. ## experiment and results our model was implemented using pytorch @xcite , and transformers from hug-gingface @xcite . the model was trained for 10 epochs on an nvidia rtx-a6000 gpu, with a batch size of 32 and a learning rate of 1 × 10 -3 . we used the officially provided training, validation, and test splits to train, evaluate, and test our model's performance. we have evaluated the performance of our model based on precision, recall, and micro-f1 score, as mentioned in the shared task guideline. the experimental results are summarized in our experiments showed that the base arabertv0.2-twitter (antoun et al., 2020) outperformed arabertv0.2. additionally, the best-performing model was the combination of arabertv0.2-twitter with conv1d, bigru, and attention, which achieved an f1 score of 89.32% on the evaluation set, highlighting the benefits of combining these techniques for this specific task. as shown in ## discussion our experimental results demonstrate that integrating a bert-based model with cnns, bigru, and an attention mechanism in a two-channel network enhances the base model's performance. the ablation studies confirm that each component contributes to improved outcomes, validating the robustness of our architecture. moreover, the arabertv0.2-twitter model outperforms the arabertv0.2 model, suggesting that pretraining on twitter data more effectively captures the structural nuances of social media text, which aligns better with the wojoodner 2024 corpus. future research should explore multi-head models and address low-represented classes to further enhance performance. focusing on these areas could provide deeper insights and improvements in arabic named entity recognition (ner). ## conclusion in this paper, we introduced the architecture we used for the ner shared task 2024, subtask-1. our model is based on a bert language model and uses a two-channel parallel neural network with an attention mechanism. our architecture combines a convolutional layers channel and a bidirectional gru channel to extract local and long-range dependencies within sentences. we found that the attention mechanism further improves the model's performance by allowing the model to focus on relevant parts of the input sequence. as shown in our ablation studies, all components of the model contributed to enhancing the final results. our model achieved a micro-f1 score of 90.13% on the test set for subtask-1 of wojoodner 2024 shared task, securing the second position.