NERD z: A Preliminary Dataset of Named Entities for A lgerian (2022)

Abstract: This paper introduces a first step towards creating the NERDz dataset. A manually annotated dataset of named entities for the Algerian vernacular dialect. The annotations are built on top of a recent extension to the Algerian NArabizi Treebank, comprizing NArabizi sentences with manual transliterations into Arabic and code-switched scripts. NERDz is therefore not only the first dataset of named entities for Algerian, but it also comprises parallel entities written in Latin, Arabic, and code-switched scripts. We present a detailed overview of our annotations, inter-annotator agreement measures, and define two preliminary baselines using a neural sequence labeling approach and an Algerian BERT model. We also make the annotation guidelines and the annotations available for future work

## introduction named entity recognition (ner) is one of the most fundamental tasks in information extraction, and natural language processing in general. resources for ner have been largely developed for several languages. despite recent advances in machine learning and cross-lingual approaches, manually annotated corpora for individual languages remain a prerequisite to achieve high accuracy @xcite . this is especially true for small, underresourced languages and dialects. in this work, we focus on the vernacular algerian language, a non-standardized spoken arabic variety, characterized by heavy use of code-switching and borrowings. it is a morphologically-rich, noncodified, spoken semitic language @xcite , and can be written in both arabic and latin scripts. arabic varieties written in latin script are referred to as arabizi, and likewise narabizi is used to refer to the north african arabizi forms @xcite . we will therefore, in what follows, refer to algerian written in latin script as narabizi. we also make a distinction for algerian written in arabic script, and refer to it in what follows as alg-arabic. the non-standardization of algerian is indicated by a high variance in morphology, phonology, and lexicon. a word can be written in different ways both in narabizi and alg-arabic scripts. arabic phonemes that do not exist in the latin alphabet, are usually substituted by digits that are visually similar to the arabic letter @xcite . despite not being standardized, algerian is extensively used online and on social media. the amount of algerian resources does however not reflect its widespread use. algerian is underresourced, and few annotated corpora are available. one of the most recent and most valuable resources for algerian is the manually annotated narabizi treebank @xcite , and it's extended version that includes transliterations to alg-arabic and code-switched scripts @xcite . we use this dataset of user-generated corpus that reflect the non-standardized nature of the algerian vernacular, and annotate it for named entities. in this work, we present nerdz a preliminary and first publicly available dataset of named entities for the vernacular algerian dialect. the annotations of entities are added on top of the extended narabizi treebank @xcite , where each sentence of the narabizi treebank is manually transliterated into arabic script and a code-switched version. nerdz therefore contains parallel entities written in both latin and arabic scripts. in addition, we provide some preliminary baseline results based on a neural architecture for ner that combines character-level cnn, word-level bilstm, and a crf inference layer. in section 2, we give a brief description of the narabizi treebank by @xcite , and it's extended annotations by @xcite . in section 3 we describe the nerdz dataset, the the examples are selected from the annotations of @xcite . the translation to english is added for readers' comprehension. the nerdz dataset builds on the extension of the narabizi treebank @xcite , by adding named entity annotations. the narabizi treebank 2 contains manually annotated syntactic and morphological information, and comprises around 1,500 sentences. these are mostly comments from newspapers' web forums (1,300 sentences from @xcite ), in addition to 200 sentences from song lyrics. the sentences are annotated on five different levels, covering tokenization, morphology, identification of code-switching, syntax, and translation to french @xcite . @xcite have further extended the narabizi treebank, by first cleaning the treebank for duplicates, correcting some of the french translations, and some of the code-switching labels. but most importantly, they manually transliterated each sentence into purely alg-arabic and code-switched scripts. the treebank therefore has three parallel writing forms for each token in a sentence. due to the preprocessing, this version of the treebank @xcite is a little bit smaller than the original treebank @xcite . ## annotations of named entities in nerdz the named entity annotations in nerdz are continuous, non-overlapping, spans of strings. the string boundaries follow the tokenization in the narabizi treebank @xcite , where each token is assigned one entity type. unfortunately, the narabizi treebank has a lack of consistency in the tokenization. for example the definite article "el" can be found both as a single token, and attached to a token. this is an issue that should be addressed, however, we did not correct the tokenizations in this work. fixing tokenization will alter the dependency trees, and our annotators were not trained to perform this task. for our annotations, we use the web-based anno-tation tool brat @xcite . nerdz is annotated using the iob2 scheme for eight entity types: per, gpe, org, norp, evt, loc, prod, and misc. our annotation guidelines are partly based on the ace @xcite , conll @xcite , and ontonotes @xcite datasets. where each entity type is defined as follows: two native speakers annotated all sentences from the narabizi treebank. to start with, the annotators selected a random sample of 100 sentences that they annotated together. this was done to settle on the type of entities to annotate, and to define the annotation guidelines. once this was clarified, each annotator annotated the entire treebank. it is for this round of annotations that we computed the inter-annotator agreement. we compute two measures of agreement, krippendorff's alpha and micro f1-score. in terms of krippendorff's alpha, the agreement score is α = 0.87, which suggests strong evidence for good agreement. the agreement in terms of micro f1-score achieved 86.3. this evaluation score is based on semeval 2013 task 9 evaluation scheme foot_1 @xcite . here, we used the strict measure, and compute f1 for exact match of both the entity boundary (the span of the entity), and the entity type. we disregard all annotations where both annotators agree that a token is not an entity, i.e., the o tag. for our experiments, multiple annotations i.e., annotations with disagreements, were subsequently discussed by both annotators until agreement, and one anno- example 1 example 2 example 3 token annotation token annotation token annotation l b-org el b-org -khadra i-org khadra i-org alkhadra b-org figure 1 shows the confusion matrix of the annotations. the annotators have a high agreement for the entity types per, gpe, org, and norp with respectively an achieved f1-score of 96.0, 91.6, 87.2, and 80.3. however, there is some disagreement for the types org and norp. a close analysis of this showed that the main problem here is the span of annotations. as previously mentioned, the narabizi treebank has no consistency in tokenization. despite the annotators agreeing on for example when the definite article "el" should be part of an entity or not, it is clear that the tokenization has influenced their choices. as algerian is non-standardized, the definite article "el" can also be written as "al" or "l", which is not always tokenized correctly. ## experimental setup, results, and analysis we use two preliminary benchmarks: an ncrf++ (yang and zhang, 2018) model, and we fine-tune the algerian bert model dziribert @xcite for the ner task. ncrf++ is a pytorch framework for neural sequence labeling. our model is similar to previ-ous state-of-the-art models for english and norwegian @xcite @xcite , and is a combination of character-level cnn, word-level bilstm, and a final crf layer. the word-level bilstm takes as input a concatenation of character representations from the cnn and pre-trained word embeddings. we use the fasttext algerian embeddings used by @xcite , and which were trained on a large user-generated algerian code-switched dataset @xcite . we use the implementation of dziribert that is made available via the huggingface library @xcite , and fine-tune it for ner using our dataset. we ran three baselines, for each of our annotated scripts: narabizi, alg-arabic, and code-switched. we use the same fixed random seed in all of our experiments, and keep the ncrf++ parameters on their default values foot_2 . for dziribert we use a learning rate of 5e-3, and train for 5 epochs. following the semeval 2013 task 9 evaluation scheme @xcite , our evaluation uses f1-score with strict strategy: exact boundary and entity type. the first observation is that the ncrf++ model constantly outperforms the dziribert model. ncrf++ performs best on the code-switched version of the data, while dziribert is better on the alg-arabic script. this we believe is dues to the data present in the embeddings used with ncrf++, and the data used to train dziribert. both models perform worst on the narabizi script, which constituted most out-of-vocabulary words in the embeddings used with ncrf++ (95.95% for narabizi, compared to 22.02% for alg-arabic, and 33.36% for code-switched). to investigate this further, we ran the same experiments on the four most frequent entity types, namely per, org, gpe, and norp, and removing the other non-frequent entities. as ncrf++ yielded the best results, we will focus on this benchmark for this analysis. the results of the entity typelevel for dziribert can be found in appendix a, in tables and 8 and 9 . from ## conclusion and future works we present our annotations to expand the narabizi treebank @xcite tity annotations. the released preliminary dataset, nerdz, is the first publicly available ner dataset for algerian, including parallel entities written in latin and arabic scripts. we also provide two simple benchmark experiments on the three scripts of the datasets latin, arabic, and code-switched. despite its current small size, nerdz is a richly annotated dependency treebank. this is a preliminary version of the dataset, in future work we plan to expand the size of the dataset by using the 8,673 sentences from @xcite not included in the narabizi treebank. we plan to update the annotation guidelines to include nested entities which might reduce the disagreement between annotators. we also plan to experiment with more models, and compare our baselines to e.g., cross-lingual ner approaches. we would also like to look further into tokenization and embedding related issues.