CA ra NER : The COVID -19 A rabic Named Entity Corpus (2022)

Abstract: Named Entity Recognition (NER) is a well-known problem for the natural language processing (NLP) community. It is a key component of different NLP applications, including information extraction, question answering, and information retrieval. In the literature, there are several Arabic NER datasets with different named entity tags; however, due to data and concept drift, we are always in need of new data for NER and other NLP applications. In this paper, first, we introduce Wassem, a web-based annotation platform for Arabic NLP applications. Wassem can be used to manually annotate textual data for a variety of NLP tasks: text classification, sequence classification, and word segmentation. Second, we introduce the COVID-19 Arabic Named Entities Recognition (CAraNER) dataset. CAraNER has 55,389 tokens distributed over 1,278 sentences randomly extracted from Saudi Arabian newspaper articles published during 2019, 2020, and 2021. The dataset is labeled by five annotators with five named-entity tags, namely: Person, Title, Location, Organization, and Miscellaneous. The CAraNER corpus is available for download for free. We evaluate the corpus by finetuning four BERT-based Arabic language models on the CAraNER corpus. The best model was AraBERTv0.2-large with 0.86 for the F1 macro measure.

## introduction named entity recognition (ner) is a classical sequence classification problem where each word in a given sentence is assigned to one of a predefined list of tags such as person name ( shaker, biden), location ( cairo, washington), and organization ( united nations, al-ittihad club). ner is a key component and a fundamental task for many nlp applications, including information extraction @xcite , question answering @xcite , content recommendations @xcite , customer support @xcite , and information retrieval @xcite . it is one of the earliest tasks of nlp using classical statistical algorithms such as maximum entropy @xcite and has been developed for many years. however, it is still relevant in the current time where we are using transformer-based language models such as bidirectional encoder representations from transformers (bert) @xcite . despite the recent advances in the nlp systems due to the usage of deep learning models, especially transformer-based language models, the need for new annotated datasets for developing ner systems is still crucial, where each domain and application requires its own dataset and tags. in general, ner systems, from our perspective, face three challenges: first, the widespread use of nlp applications in different domains necessitates the usage of texts from these domains, which are probably different in their genre, style, and vocabularies, from the available annotated ner datasets. out-ofvocabulary (oov) will be the first challenge the ner system will face. for instance, if we need high-performance ner models, a ner dataset for the legal domain can not be used for the medical domain, and a ner dataset for moroccan newspapers will not be the best choice for ner applications for uae newspapers. second, unlike fixed tagset applications such as part-of-speech tagging or word segmentation, each ner application requires different tagsets. most of the ner available datasets concentrate on person names, location, and organization tags with slight differences among them on other tags, such as the availability of geopolitical entities tags for government entities such as the ontonotes 5 ner dataset @xcite . consider the need for a ner dataset for a food delivery chatbot; in this case, we may need a tagset containing tags for: a) the person's name to know who ordered the food, b) different tags for food items to direct the order to a relevant restaurant, and c) address to know the delivery location. or suppose a system to analyze newspaper articles for military clashes; such a system, in addition to time, location, and the number of injuries, will need different tags to identify different kinds of weapons, for example. third, even if there is a dataset for particular domains or genres, there are always new topics, interests, and concepts introduced to those domains and genres that may degrade the models trained on older datasets. such a challenge is well known in the machine learning community as data and concept drift @xcite @xcite . consider, for instance, a ner system trained on annotated texts from newspapers during the 2000s and then applied to newspaper texts during the covid-19 pandemic; will this system perform well? the contribution of this paper is in three folds. first, we introduce wassem ( in arabic, "annotate" in english), a platform for arabic textual data annotation based on the django framework. second, we used wassem to prepare covid-19 arabic named entity recognition dataset (caraner): a ner dataset annotated with six tags (person, organization, location, title, miscellaneous, and other) covering 1,278 sentences, randomly extracted from saudi arabian newspapers part of arabic newspapers covid-19 corpus (aranpcc) @xcite . the covid-19 part of the corpus name "caraner" is a temporal reference to the covid-19 period as the aranpcc corpus covers one year before the covid-19 pandemic and two years after the emergence of the pandemic (i.e. 2019 -2021). third, using caraner, we evaluate four bertbased arabic language models, namely bert-basemultilingual-cased @xcite (baseline), arabertv0.2-large (antoun et al., 2020), camelbert-msa @xcite , and gigabert-v4 @xcite . the rest of this paper is organized as follows: in section 2, we review the related work on arabic ner. we briefly describe the main building blocks for wassem in section 3. section 4 describes the process of the caraner dataset construction and annotation and its basic statistics. section 5 illustrates and discusses the result of fine-tuning four language models using caraner. we conclude the paper in section 6. ## related work previous works on ner can be divided into two categories: building named-entity-tagged corpora and building ner models. in this section, we focus on previous work on building named-entity tagged arabic corpora. previous works on building namedentity corpora cover a variety of languages, genres, and domains. these studies focused on many tag sets that differ according to the application and domain requirements. some corpora in the literature cover general-purpose tag sets from broad domains such as newswire and wikipedia. in contrast, others focus on specific tag sets, such as the medical domain. most previous studies include the four named-entity tags: person, location, organization, and miscellaneous. the interest in building arabic ner corpora dates back to the 2000s. one of the earliest studies for building an arabic named-entity annotated corpus is the ace 2004 multilingual training corpus @xcite . the ace 2004 corpus is developed by ldc and contains text in arabic, chinese, and english, covering a variety of genres. it was annotated for many nlp tasks, including named entity recognition and relation extraction. the ace 2004 entity tags are person (per), geo-political entity (gpe), organization (org), and facility (fac). the size of the arabic portion of ace 2004 is around 10k tokens and collected from newswire texts. another ldc-licensed multilingual corpus is ontonotes 5 @xcite , which is collected from various genres, including newswire and conversational telephone speech in three languages: arabic, chinese, and english. the arabic portion of ontonotes 5 contains around 300k tokens. similar to our corpus, the arabic ontonotes 5 corpus was collected only from newswire sources in modern standard arabic (msa) and annotated with 18 entity types. the anercorp corpus @xcite ) is collected from modern standard arabic media texts. it contains around 150k tokens tagged with four entity types: person, organization, location, and miscellaneous. for genres other than newswire, mohit et al. the most important factor that may distinguish the caraner dataset is that it was sampled from the covid-19 period. regarding the caraner size (∼50k tokens), we are working to increase its size to reach a level that can produce good results using state-of-the-art machine learning algorithms, specifically neural language models. ## annotation platform the motivation behind the development of wassem is the shortage of open source annotation platforms suitable for arabic nlp annotation tasks such as word segmentation and diacritization. we designed wassem to help in the following tasks: a) text and sentence annotation for applications such as text classification and sentiment analysis, b) sequence annotation for applications like ner and pos tagging, c) subword annotation for applications like arabic words segmentation, and d) for arabic word diacritization. wassem has four main functions, which are described as follows: four steps are needed to complete this process as follows: first, the administrator needs to define the list of tags used for the annotation task, with a brief description for each tag if they do not exist before in the database. also, the administrator can attach a list of words with fixed tags such that the corresponding tag for each word in the list does not change when the context changes; this accelerates the annotation process. for example, for pos tagging, this list may include particles and prepositions such as " " (to), " " (about), " " (but) or part of the most frequent words in the data that have the same characteristic, i.e., they have fixed tags such as " " (allah), " " (said), " " (to). the system will automatically annotate words with their corresponding tags in the list such that the user does not need to consider them during manual annotation. such lists of words and their fixed tags can be used in the future for other annotation tasks. second, the system currently provides manual annotation on the document level and word level. based on the type of annotation task, the system administrator should determine the level of the task. the difference between the two levels is the text unit that will be annotated with the tag. hence, in the case of the document level, a label will be assigned to the entire sentence/text, for example, sentiment analysis on the document level. in contrast, for the word-level annotation, each word/token in the text will be labeled with a tag, for example, pos tagging. moreover, for the case of word-level annotation, the administrator should specify if the task is a segmentation, diacritization, or tagging the whole token (e.g., ner). third, the administrator needs to provide a description of the annotation tasks and identify the minimum number of annotators who can participate in the task. the system can automatically assign a final label using the majority vote if there are three or more annotators. fourth, the administrator should upload the raw data that will be tagged if it was not in the system before (i.e., used previously on other annotation tasks), and link this annotation task with the appropriate list of tags. b. annotation task assignment: after creating the annotation task, the administrator should assign it to the annotators. the administrator can add new annotators or select annotators already existing in the system. wassem's website provides a link to a registration form for volunteer annotators where they need to provide their contact information, gender, age, and educational background. such information will help the administrator to identify the best annotators for each annotation task. if there is no agreement (i.e., tie), the final tag will be set as "no_agr". in the case of uncompleted tasks (i.e., some annotators have not completed their tasks yet), the system will set the tag as "not_annot" to the examples which are not annotated yet. ## data in this section, we describe our work to prepare the raw data for the annotation process, the tagset used for annotation, the annotators' training and annotation process, and the final data after annotation. we made the dataset is available for free download on github foot_0 . ## evaluation ner is usually treated as a sequence labeling problem. in the literature, early studies used crf models @xcite . later, deep learning sequence models such as lstms have been used in many studies for ner @xcite . more recently, pre-trained language models have been used to model ner, and they outperform previous models (yohannes and amagasa, 2022). arabertv0.2-large language model outperforms the other models followed by camelbert-msa. the superiority of the arabert over other models can be explained by the fact that arabertv0.2large is much larger than the other models. in particular, arabertv0.2-large has 371m parameters, whereas the other models are based on the smaller model bert-base, which has less than half this number of parameters. however, we observe that camelbert-msa achieved a comparable performance by only using less than half of the model size. this relatively good performance of camelbert-msa can be attributed to the size of the data on which the model was trained compared to the other models. from these results, we observe that all models achieved an accuracy score of more than 95%. this can be attributed to the fact that most of the words have an "o" tag, which makes it easy to achieve such a high accuracy score. in particular, only 3,813 (∼7%) out of 55,389 tokens are named entities, and 51,576 (∼93%) of the tokens are not. ## conclusion in this paper, we introduced a web-based annotation platform for arabic nlp (wassem) and a new dataset for arabic ner (caraner) annotated with five tags (per, tit, loc, org and mis) using wassem. experimentation on four bertbased language models shows that fine-tuning arabertv0.2-large on caraner gives the best results among the other models, with a 0.86 macro f-1 score. also, the relatively good performance of camelbert-msa (0.84 macro f-1 score) may suggest that using large and diverse datasets for pretraining smaller language models (i.e., bert-base) gives similar performance to larger models (i.e., bert-large) pre-trained on smaller datasets. in the future, we plan to double the size of caraner to improve the performance and experiment with different arabic language models.