A Semi-Supervised BERT Approach for A rabic Named Entity Recognition (2020)

Abstract: Named entity recognition (NER) plays a significant role in many applications such as information extraction, information retrieval, question answering, and even machine translation. Most of the work on NER using deep learning was done for non-Arabic languages like English and French, and only few studies focused on Arabic. This paper proposes a semi-supervised learning approach to train a BERT-based NER model using labeled and semi-labeled datasets. We compared our approach against various baselines, and state-of-the-art Arabic NER tools on three datasets: AQMAR, NEWS, and TWEETS. We report a significant improvement in F-measure for the AQMAR and the NEWS datasets, which are written in Modern Standard Arabic (MSA), and competitive results for the TWEETS dataset, which contains tweets that are mostly in the Egyptian dialect and contain many mistakes or misspellings.

## introduction in recent years, researchers have become increasingly interested in developing deep learning solutions for arabic natural language processing applications. arabic is considered one of the most spoken languages in the world. however, compared to any non-arabic language such as english, it is considered a much more challenging language because of its high ambiguity and rich morphology. in this paper, we tackle the problem of arabic named entity recognition (ner) using a semisupervised learning approach. ner is the task of extracting, locating, and classifying named entities in a given piece of text. the named entity can be a proper noun, a numerical expression representing type unit or monetary value, or a temporal value that represents time. in this work, we focus on recognizing proper nouns only and classifying them into one of three classes: a person, a location, or an organization in a bio (beginning, inside, outside) format. ner is a particularly difficult task for arabic. first, there is no capitalization in the arabic script, commonly used in non-arabic languages such as english to detect named entities. second, arabic can be ambiguous; for instance, a lot of named entities are also used as common nouns and adjectives. arabic is also known for its rich morphology. finally, one major issue that hinders arabic nlp research, including ner, is the lack of sufficient resources. such resources include arabic corpora and gazetteers that can be leveraged to perform the nlp tasks. even if some of these resources are present, they are usually limited in scope or not publicly available. to overcome the aforementioned challenges related to arabic nlp, we propose a semi-supervised deep learning approach for arabic ner inspired by the work of @xcite . the idea is to train two bert-based models: a teacher model and a student model. bert stands for bidirectional encoder representations from transformers. the bert teacher model is trained on a small labeled data set and then applied on a huge semi-labeled dataset to predict the classes of its unlabeled tokens. the output is then used to train a student model with the same architecture as the teacher model, and then the student model is fine-tuned using the small labeled dataset used to train the teacher model. to evaluate our approach, we used three different arabic ner benchmarks, namely aqmar @xcite , news @xcite and tweets @xcite . we compared our approach to various baselines and state-of-the-art ner tools and we outperformed all of them in the case of aqmar and news datasets and achieved comparable performance in the case of the tweets dataset. the paper is organized as follows. in section 2, we review the related work. section 3 describes our semi-supervised learning approach. in section 4, we evaluate our proposed approach on different datasets. finally, we conclude and present future directions in section 5. ## related work many approaches have been proposed in the literature to perform arabic ner. these approaches can be categorized into three main categories: machine-learning-based approaches, rule-based approaches, and hybrid approaches. in a survey on arabic nlp @xcite , the authors reviewed a set of machine-learning-based arabic ner approaches. some approaches utilized conditional random fields (crf) @xcite @xcite , while others relied on supportvector machines (svm) @xcite @xcite . other approaches relied on meta-classifiers @xcite @xcite . all these approaches utilized different combinations of features such as lexical, contextual, morphological, gazetteer, syntactic and pos features. to date, there are a few works that studied deep learning for the task of arabic ner. gridach @xcite utilized character-level neural networks and conditional random fields, in a fully-supervised fashion. however, this approach was trained and tested using only one dataset and was not evaluated on multiple datasets as in our case to assess its generalization capabilities. helwe and elbassuoni @xcite proposed a semi-supervised learning approach based on an algorithm called co-training, which was adapted to the context of deep learning for the task of arabic ner. their method makes use of a small amount of labeled data, which is augmented with partially labeled data that is automatically generated from wikipedia. their model is based on an ensemble of two bi-lstms. we used the same training, validation, and testing datasets from @xcite to evaluate our approach. antoun et al. @xcite pre-trained a bert model for arabic called arabert, which was evaluated on different tasks such as sentiment analysis and ner. in our approach, we used their pre-trained model and re-trained it in a semi-supervised fashion for the task of arabic ner. many rule-based approaches have been proposed for arabic ner. most of these approaches relied on different combinations of features including lexical triggers @xcite , morphological analyzers @xcite @xcite , regular expressions and gazetteers @xcite , and transliteration @xcite . most of these reviewed methods however were trained and tested using very limited data, typically less than a hundred documents, thus it is not clear how well they can generalize to other datasets. moreover, none of these approaches were evaluated on any established benchmarks for the task of arabic ner. the only exceptions are the approaches by @xcite , which were trained and tested on the automatic content extraction (ace) @xcite and the treebank arabic datasets, and the approach by elsebai et al. @xcite , which was trained and tested using more than 500 news articles. another type of approaches commonly used for ner is the hybrid approaches, which combines machine-learning-based and rule-based techniques such as @xcite @xcite . the advantage of our approach over the above mentioned approaches is that we build a more robust machine-learning-based model by training a bert neural network in a semi-supervised fashion using fully labeled and semi-labeled datasets. we compared our approach to the state-of-the-art approaches (i.e,. those with the highest reported performance from the list above, namely madamira @xcite , farasa @xcite and deep co-learning @xcite ) and outperformed them on two different msa datasets. ## approach to train a robust model for arabic ner using deep learning, a sufficiently large training data is needed. given the lack of such data in the case of arabic, we propose a semi-supervised learning approach. our approach is based on a teacher-student learning mechanism inspired by @xcite . it relies on two datasets for training: a fully labeled dataset and a partially labeled dataset. each instance of these datasets is a sentence composed of word tokens and their labels (person, organization, location or other) if they exist. figure 1 shows an example instance of the fully labeled dataset. as can be seen from the figure, every token is associated with a label. figure 2 shows an example instance of the partially labeled dataset, where some of the tokens are labeled and some are not. the core model in our approach is a pre-trained arabic bert model called arabert. in brief, our semi-supervised approach works as follows: a bert teacher model is trained on the fully labeled training dataset to classify the non-labeled tokens of the partially labeled dataset. the best instances from these weakly labeled sentences are then chosen to train a bert student model, which will be later fine-tuned using the fully labeled training dataset. in the remaining of this section, we first describe the pre-trained arabert model. we then describe our proposed semi-supervised learning approach for arabic ner. ‫ر‬ ‫ا‬ ‫و‬ ‫او‬ ‫ه‬ ‫ر‬ ‫ا‬ ‫ان‬ ‫وا‬ ‫ر‬ ‫ذ‬ ‫ا‬ ‫ل‬ ‫ا‬ ‫ا‬ b-loc b-loc b-org i-org figure 2: instance of the partially labeled dataset 3.1 arabert model arabert is a pretrained arabic language model developed by antoun et al. (antoun et al., 2020) based on a transformer architecture called bert. the bert model consists of a stack of transformer blocks which was pre-trained on two tasks: masked language modeling (mlm) and next sentence prediction (nsp). the task of mlm consists of training the model to predict a masked word given the other words in a sentence. the task's dataset is constructed by choosing 15% of its tokens to be masked by replacing: 80% with the [mask] token, 10% with a random token, and 10% with the original token. while the task of nsp consists of training the model to learn the relationship between two sentences by taking as input two sentences a and b and predicting if sentence b follows sentence a. the arabert model was pre-trained on a large dataset of 70m arabic sentences with 3b words. the training data was collected from different publicly available corpora such as the arabic wikidumps, the 1.5b words arabic corpus, the osian corpus, and a corpus of assafir news articles. in addition to the publicly available datasets, the authors augmented the dataset by manually crawling news websites such as al-akhbar, annahar, al-ahram, and al-wafd. ## conclusion this paper presented a new approach to detect and classify named entities in any arabic text. our approach consists of training an already pre-trained bert model for arabic ner in a semi-supervised fashion. we made use of two datasets. the first dataset was fully labeled, while the second dataset was partially labeled. we evaluated our approach on three datasets. it outperforms all other arabic ner tools and approaches on two testing datasets, namely news and aqmar datasets. for the tweets dataset, helwe and elbassuoni's deep co-learning approach (helwe and elbassuoni, 2019) scores a higher fmeasure than our method because the bert model was pre-trained and trained on mainly msa corpora that do not contain mistakes and misspellings. in future work, we plan to pre-train the bert model on tweets to make it more suitable for text that could contain misspellings and mistakes and which is not necessarily written in msa. we believe that this will result in an improved performance of our approach on the tweets datasets. finally, we plan to apply our semi-supervised bert-based learning approach to other nlp tasks such as part-of-speech tagging and dependency parsing.