A lex- U 2023 NLP at W ojood NER shared task: A ra BINDER (Bi-encoder for A rabic Named Entity Recognition) (2023)

Abstract: Named Entity Recognition (NER) is a crucial task in natural language processing that facilitates the extraction of vital information from text. However, NER for Arabic presents a significant challenge due to the language’s unique characteristics. In this paper, we introduce AraBINDER, our submission to the Wojood NER Shared Task 2023 (ArabicNLP 2023). The shared task comprises two sub-tasks: sub-task 1 focuses on Flat NER, while sub-task 2 centers on Nested NER. We have participated in both sub-tasks. The Bi-Encoder has proven its efficiency for NER in English. We employ AraBINDER (Arabic Bi-Encoder for Named Entity Recognition), which uses the power of two transformer encoders and employs contrastive learning to map candidate text spans and entity types into the same vector representation space. This approach frames NER as a representation learning problem that maximizes the similarity between the vector representations of an entity mention and its type. Our experiments reveal that AraBINDER achieves a micro F-1 score of 0.918 for Flat NER and 0.9 for Nested NER on the Wojood dataset.

## introduction named entity recognition (ner) is a fundamental task in natural language processing that involves identifying and classifying named entities, such as person names, locations, organizations, and temporal expressions, within text. in recent years, deep learning models, particularly transformer-based architectures @xcite , have revolutionized ner by capturing contextual information effectively. however, applying these models to arabic ner presents several difficulties. one of the major challenges is the lack of comprehensive and annotated arabic ner data, which hinders the fair evaluation of arabic ner models @xcite ., * * equal contribution 2023 ). other previous work addressed named entity recognition as a sequence labeling problem @xcite , span-based classification @xcite or seq-to-seq generation @xcite . there have been some approaches that dealt with the problem as a machine reading comprehension problem (mrc) @xcite . meanwhile binder @xcite deals with ner as a representation learning problem that maximizes the similarity between the vector representations of an entity mention and its type. this makes it easy to handle nested and flat ner alike, and can better leverage noisy selfsupervision signals. moreover, it demonstrates superiority over past approaches in terms of speed and efficiency. the use of dual networks dates back to @xcite for signature verification and @xcite for face verification. moreover, @xcite conducted a comparison of three distinct architectures bi-encoder, poly-encoder, and cross-encoder all employing deep pre-trained transformers as encoders. in our solution, we use the bi-encoder architecture that has also been used in various tasks, such as information retrieval @xcite , open-domain question answering @xcite , and entity linking @xcite and proved to achieve state of the art results. furthermore, in recent work, all tokens or spans that do not represent entities (non-entities) were categorized under a single class called "outside" (o). notably, our solution diverges from this conventional method since we use the proposed dynamic thresholding loss within the context of contrastive learning. this approach involves learning dynamic thresholds specific to candidates, aiding in the differentiation of entity spans from non-entity ones. while contrastive learning (cl) has considerably advanced numerous natural language processing (nlp) tasks, its application within the arabic con-text has been somewhat limited @xcite . recently, there has been a focus on this area as @xcite demonstrated the efficacy of cl for arabic hate speech detection, resulting in significant improvements over baselines. in a similar vein, @xcite conducted experiments applying cl to diverse arabic nlp tasks including dialect identification, emotion classification, sarcasm detection, and the identification of abusive and adult content. in this paper, we bridge the gap by introducing arabinder, a novel approach to address these challenges. binder @xcite , learns to differentiate between entities and non-entities, even when confronted with limited annotated data. this capability enhances its generalization potential, rendering it applicable to both nested and flat ner paradigms. for sub-task 1 and sub-task 2, we apply arabinder using our best model achieving micro f1 scores of 0.918 and 0.90 respectively. ## data we conducted our work on the wojood @xcite dataset provided by the shared task @xcite . the shared task focuses on identifying named entity mentions in unstructured text and classifying them into predefined classes this is divided into two sub-tasks, sub-task 1 focuses on flat ner while sub-task 2 centers on nested ner. the data for sub-task 2 differed in the manner that in the nested scheme some tokens had more than one entity type assigned to it. the corpus of wojood consists of about 27k sentences and 550k tokens and is manually annotated covering both modern standard arabic (msa) and dialect arabic (da) in multiple domains. it contains about 75k entities, out of which 22.5% are nested. the data was annotated for 21 entity types with iob tags. the dataset introduced four new tags which are occupation, website, unit, and currency. we follow the data split provided by the shared task: 70% of the data for training, 10% for development, and 20% for testing. ## method in this section, we introduce the methodology of arabinder, which utilizes the bi-encoder architecture first introduced in @xcite for arabic-named entity recognition (ner). the foundation of our model is the bi-encoder framework, which involves encoding both entity types and text using the transformer-based architecture. to provide a comprehensive understanding, we begin by explaining the background of this bi-encoder framework. by leveraging the bi-encoder architecture and incorporating contrastive learning objectives, arabinder presents a robust and effective approach for arabic ner. in the following sections, we will elaborate on the implementation details and experimental results to validate the performance of our proposed methodology. ## results in all our experiments, we exploit the arabertv2-twitter base that is trained on msa in addition to multi-dialect arabic tweets, since our data contain both msa and da in multiple domains. we demonstrate our results on the development set for flat ner and nested ner in ## discussion as can be shown from tables 2 and 3, the model performs better for nested ner than for flat ner on the development set. we noticed this behavior in several experiments. however, as can be seen from tables 4 and 5, it performs better for flat ner on the test set, this indicates that the model may have failed to generalize. in the path forward, our focus will revolve around enhancing the performance of underperforming nested experiments while delving into the exploration of alternative encoders for arabic, such as jaber @xcite , which could potentially enhance our results. moreover, we are dedicated to further refine our data pre-processing strategies to tackle the unique challenges posed by arabic, rectifying annotation errors, and addressing the scarcity of precise data. ## conclusion in this paper, our approach revolves around the application of arabinder to tackle both flat and nested named entity recognition (ner) tasks within the shared context. this methodology involves using a bi-encoder architecture, proficiently encoding both entity types and textual content. the infusion of contrastive learning into this framework serves to maximize the similarity between individual entity types and their corresponding mention spans. our evaluation revolved around bert-based models trained on arabic corpora, with a special focus on arabert. through assessment, we observed that the arabertv2-twitter base, pretrained on arabic data encompassing modern standard arabic (msa) and twitter data, performed the best. notably, it performed better for the flat ner task, outperforming its nested ner counterpart.