Cross-Dialectal Named Entity Recognition in A rabic (2023)

Abstract: In this paper, we study the transferability of Named Entity Recognition (NER) models between Arabic dialects. This question is important because the available manually-annotated resources are not distributed equally across dialects: Modern Standard Arabic (MSA) is much richer than other dialects for which little to no datasets exist. How well does a NER model, trained on MSA, perform on other dialects? To answer this question, we construct four datasets. The first is an MSA dataset extracted from the ACE 2005 corpus. The others are datasets for Egyptian, Morocan and Syrian which we manually annotate following the ACE guidelines. We train a span-based NER model on top of a pretrained language model (PLM) encoder on the MSA data and study its performance on the other datasets in zero-shot settings. We study the performance of multiple PLM encoders from the literature and show that they achieve acceptable performance with no annotation effort. Our annotations and models are publicly available ( https://github.com/niamaelkhbir/Arabic-Cross-Dialectal-NER ).

## introduction the arabic language, encompassing classical arabic (ca), modern standard arabic (msa), and various dialects of arabic (da), stands out for its linguistic diversity and intricate morphology. this linguistic complexity presents a unique challenge for natural language processing (nlp) tasks, particularly in the field of named entity recognition (ner). modern standard arabic serves as the formal reference, and many research efforts have been dedicated to msa ner. the literature on msa ner methods has witnessed an evolution from rule-based methods, to machine learning models based on hand-crafted features and subsequently deep learning models incorporating rich contextual representations. notably, pretrained transformer-based language models have recently driven significant advancements in arabic ner. arabic, however, has more than 20 distinct dialects and around 100 regional variants, which are widely used in everyday communication, particularly in digital spaces. this emphasizes the urgent need for nlp models capable of effectively handling this linguistic diversity. however, these dialects exhibit significant linguistic variation, including differences in spelling, morphology, and syntax, making it exceptionally challenging to develop a unified global modeling approach. additionally, there is no standardized spelling for these dialects. in addition, the scarcity of annotated dialectal data has been a major obstacle to progress in the field of dialectal ner. our research is driven by the goal of bridging the linguistic gap between msa and arabic dialects, specifically in the context of entity recognition. given the substantial time required for the annotation process and leveraging the success of cross-lingual transfer learning, our work focuses on exploring knowledge transfer in the context of ner, transferring knowledge from msa to various dialects. our contributions in this article are two-fold: ## dataset and annotation in this section, we introduce our datasets for modern standard arabic and arabic dialects (moroccan, egyptian, syrian), their construction, and annotation guidelines. ## task definition and model in this study, we opted to work with three distinct arabic dialects: moroccan, egyptian, and syrian. for each dialect, we selected randomly 500 sentences from the xp3x corpus and tokenized them by whitespaces before presenting them for annotation. notably, our annotation process was overseen by a single annotator, a proficient moroccan arabic speaker, with a deep understanding of egyptian and syrian dialects as well. the limited dataset size made the use of a single annotator optimal, as this approach ensured consistency, coherence, and a manageable workload, minimizing inter-annotator discrepancies and maintaining unified annotation styles. in this study, we chose to investigate three distinct arabic dialects: moroccan, egyptian, and syrian. we randomly selected 500 sentences from the xp3x corpus for each dialect and tokenized them using whitespace. our annotation process, carried out using label studio as the annotation tool, was supervised by a single proficient annotator, fluent in moroccan arabic and possessing a strong grasp of egyptian and syrian dialects. given the limited dataset size, employing a single annotator was advantageous for maintaining consistency, coherence, and manageable workloads, thereby reducing inter-annotator discrepancies and ensuring uniform annotation styles. after the annotation process, we only retained sentences containing entities for our experiments. for a comprehensive overview of the dataset's statistics, please consult tables 1 and 2 . to visualize examples from our dataset, please refer to figure 1 . named entity recognition involves identifying and categorizing named entities within text into predefined entity categories. formally, we frame the task of ner as a span classification problem. given an input sequence: @xmath0, our objective is to classify all potential spans within the sequence, defined as: here, i, j, and c correspond to the start position, end position, and span type, respectively. the probability of a specific span classification y given the input sequence x is represented as: in this equation, ϕ θ (.) is the span scoring function, and z θ (x) is the partition function. during training, our objective is to minimize the negative log-likelihood of the gold span classifications. training loss during training, our assumption allows us to bypass the need to explicitly evaluate the partition function z θ (x) to compute the loss. the loss for a single sample (x, y) ∈ t is simply the sum of loss for all spans in the input: this loss is minimized over the training set using a stochastic gradient descent algorithm. decoding during inference, our aim is to determine: in other words, we seek to identify the span labeling configuration that achieves the highest score. for unconstrained span classification, a straightforward approach is to assign the label with the highest score to each individual span, as follows: nonetheless, this decoding approach is not optimal since it may result in structural constraint violations. in our context of flat entities, overlapping entity spans are strictly prohibited. a more efficient solution, as presented in our prior research @xcite foot_1 , employs a two-stage decoding process. initially, spans predicted as nonentities are filtered out, followed by the application of a maximum independent set algorithm to the remaining spans to determine the optimal set of entity spans. ## experimental setup token encodings to encode our input tokens, we use 8 diverse pretrained language models, i.e trained on diverse dataset sources: arabic msa dataset (arbertv2 and camelbert-msa), arabic dialect dataset (marbertv2 and camelbert-da), mixture of msa and arabic dialect (arabertv2 and camelbert-mix), and multilingual dataset (mbert and mdeberta). we detail them below: hyperparameters we train all our models up to convergence. we use a training batch size of 12 and a validation batch size of 32. we employed a learning rate of 2e-5 for the pre-trained parameters and a learning rate of 3e-3 for the other parameters. we used a batch size of 8 and trained all the models to convergence (near 0 training loss). for testing, we use the last model, given the limited availability of validation data in our dataset. to manage the complexity of the task, we impose a constraint on the maximum span length, setting it to a maximum width of @xmath1. this constraint significantly reduces the number of segments from l 2 to lk. the pretrained transformer models were loaded from huggingface's transformers library, we used allennlp for data preprocessing. we trained all the models on a server equipped with v100 gpus. ## results the main results of our experiments are shown in figure 2 . we conducted two primary experiments: firstly, training on modern standard arabic, and evaluating on dialects, and secondly, reversing this configuration, training on individual dialects and assessing on msa. for both scenarios, we used the complete dataset outlined in ## conclusion and future work in this work, we explore transfer learning for named entity extraction, specifically from modern standard arabic (msa) to various arabic dialects, employing a range of pretrained language models. for this purpose, we annotated a dataset including moroccan, syrian, and egyptian dialects. our results showed that for both msa-to-dialects and dialects-to-msa scenarios, syrian data demonstrated superior performance, which suggests a robust linguistic affinity between the syrian dialect and msa. similarly, egyptian models exhibited strong results. in contrast, models trained on the moroccan dialect consistently face challenges, indicating substantial linguistic divergence between moroccan arabic and msa. in future work, we plan to include a wider range of arabic dialects to better understand the nuances and generalization of our results across different dialectal variants. in addition, we plan to explore the nested entity task.