Dynamic Ensembles in Named Entity Recognition for Historical A rabic Texts (2021)

Abstract: The use of Named Entity Recognition (NER) over archaic Arabic texts is steadily increasing. However, most tools have been either developed for modern English or trained over English language documents and are limited over historical Arabic text. Even Arabic NER tools are often trained on modern web-sourced text, making their fit for a historical task questionable. To mitigate historic Arabic NER resource scarcity, we propose a dynamic ensemble model utilizing several learners. The dynamic aspect is achieved by utilizing predictors and features over NER algorithm results that identify which have performed better on a specific task in real-time. We evaluate our approach against state-of-the-art Arabic NER and static ensemble methods over a novel historical Arabic NER task we have created. Our results show that our approach improves upon the state-of-the-art and reaches a 0.8 F-score on this challenging task.

## introduction digitized historical literature is an essential resource in facilitating historical and social research. information extraction, the task of automatically extracting structured information from digitized documents, has emerged as a method to scale the analysis of texts and allow the integration of information from different sources (e.g., @xcite , @xcite ). one of the cardinal tasks in information extraction is ner, extracting entities from text and categorizing them into predefined categories. numerous ner algorithms have been suggested, from rule-based approaches @xcite to machine learning (ml) approaches e.g., @xcite . however, an overwhelming majority of the algorithms have been developed over modern english text. rule-based approaches designed using modern english grammatical rules are irrelevant to arabic @xcite . ml approaches rely on large modern english corpora sourced from international news outlets and social media. training these tools to extract named entities from historical texts written in ancient arabic requires large amounts of tagged text in the same language and preferably the same dialect, which are sorely missing. in this work, we explore two avenues to mitigate the weakness of existing arabic ner tools over historical arabic texts @xcite . first, we utilize an approach named dynamic prediction, that was used in adjacent fields such as schema matching @xcite , business process matching @xcite , and pattern recognition @xcite . in dynamic prediction, the results of different techniques are combined according to each result's (or component of the result) predicted quality. here, we build an ensemble learning model based on dynamic prediction for ner algorithms. to alleviate the lack of resources in arabic historical ner, we have created a novel dataset -the bedaya corpus and examine how training ml-based ner tools on historic arabic impacts their performance instead of training them over modern arabic text. the contribution of this paper can, therefore, be summarized as follows. 1) we introduce a dynamic ensemble model for ner over historical arabic texts. 2) we present bedaya corpus, an arabic historical dataset. 3) we perform a detailed empirical evaluation of our approach over baseline and stateof-the-art methods. we now provide background and preliminary definitions of ner and ensemblelearning in section 2 and present related work in section 3. in section 4 we present our predictors and ensemble learning technique. in section 5, we present the datasets. in section 6, we report the results of our empirical evaluation, concluding our work in section 7. some ner algorithms provide a confidence value (usually over [0, 1]) associated with each mapping. others provide a set of confidence values for each token, one for each possible class. there are two broad approaches to constructing ner algorithms. rule based approaches, rely on rules and patterns manually created in order to capture terms from input documents. for example, mikheev et al. @xcite suggest the following rule: rule-based systems achieve high precision but require a significant time investment to develop. moreover, transferring rules from one language to another or between domains is very challenging @xcite . in machine-learning (ml)-based approaches, an algorithm containing rules or some other internal representation is learned from training examples @xcite . this approach can be further categorized into supervised, unsupervised, semi-supervised. supervised approaches rely on manually tagged examples. unsupervised approaches attempt to divide the tokens into similar groups, hopefully grouping the same class's entities. semi-supervised approaches use a small number of tagged examples to guide the otherwise unsupervised process. several features are used in ml-based approaches, such as part of speech, capitalized words, special marks (punctuation, numbers, dates, and titles), and word length. recent ner systems rely on (deep) neural-networks over sequences of words @xcite . gazetteers, or entity dictionaries, are an essential resource for ner that support entity tagging @xcite by allowing to look up words and phrases that are commonly used as named entities (e.g., @xcite , @xcite ). ner over arabic text has proved much more challenging than other languages due to the complexity of arabic morphology, absence of capital letters, and the lack of resources on which to train ml-based tools @xcite . for example, while performance on conll 2000, a common ner task in english has reached an f1-score of 97.3% @xcite , the best performance on a modern arabic corpus anercorp only achieves an f1-score of 89.9% (balla and delany, 2020). arabic texts can be categorized into classical arabic, modern standard arabic (msa), and arabic dialects @xcite . historical islamic literature is written in classical arabic @xcite . according to @xcite , msa differs from classical arabic in vocabulary, syntax and styles. moreover they present 9 common morphonology changes that happened upon the time that lead for msa from classical arabic, thus changes leads to changes in syntax and words meanings. researchers must consider these differences when using modern datasets and ner tools to build ner tools for historical arabic texts. ## related work speck and ngomo 2014 explore the use of ner ensemble learning in english, combining four ner algorithms using 15 different ensemble learning algorithms. they evaluate these methods over five different english language datasets and show that ensembles can reduce the error rate by an average of 40%. however, all of the ensemble learning methods evaluated were static methods. in this work, we employ predictors to assess the quality of ner results dynamically. for arabic ner, abdallah et al. 2012 integrate an ml-based approach with a rule-based one. they train a decision tree model to combine the results, achieving an f1-score of 87.8%. (sajadi and minaei, 2017) learn a static ensemble for named entity recognition over classical arabic text and rely on the adaboost algorithm, an implementation of the multi-class boosting method. using their novel noorcorp (5) historic arabic corpus, the system was evaluated over its ability to recognize three classes: location, person, organization. ekbal and bandyopadhyay 2010 propose a ner for arabic based on support vector machines (svm) @xcite used to classify feature vectors for every word. recent ner systems rely on (deep) neuralnetworks over sequences of words. these systems infer features from raw sentences by using several layers of components allowing to represent higher levels of abstraction over the word sequence (see survey @xcite ). the current state of the art are ner systems based on the contextual word embedding approach @xcite and especially systems which employ bert @xcite , a novel pre-trained deeply bidirectional, unsupervised language representation, that takes into account the context for each occurrence of a given word. @xcite propose an arabic ner system using a six-layer deep neural network model based on the transfer learning architectures among deep neural networks @xcite , @xcite . the system achieves an overall f1-score of 90% compared with their baseline bi-lstm-crf model which reached an overall f1-score of 73%. their work is trained and tested over msa, they used wikifane-gold @xcite as a dataset that builds over the arabic version of wikipedia. while our work focus on historical text, and emphasizes (see 6.4) the non improving ner over historical arabic text using msa. dynamic prediction has been previously proposed for schema matching @xcite and pattern matching @xcite . to the best of our knowledge, it has not been used to combine ner algorithms, as proposed here. ## employing predictors for dynamic ensemble building static ensemble methods rely on the base ner algorithms' results to generate a ner algorithm that combines their results such that on every document set, the results of the base ner algorithms are combined in the same way. predictors have been explored for schema matching @xcite and pattern recognition @xcite as a method for fine-tuning the ensemble creation method to the task at hand. predictors assess the result of each base algorithm on the task and provide a score. the ensemble model can use this score to determine whether or not the algorithm has succeeded in this specific occasion to identify the correct class. formally: definition 4.1 (predictor). let d be a set of documents and let {t d |∀@xmath0: @xmath1. let c be a set of classes, and let f : t @xmath2. let n be the set of all possible such ner algorithm results and let @xmath3, then a predictor is a function f : @xmath4. thus, a predictor assesses the quality of the algorithm's outputs without knowing the expected results. predictors are defined over different result cardinalities. dataset-level predictors output a single number for the entire dataset. sentence-level predictors output one number for each sentence, and token-level predictors output a number for each token. furthermore, as mentioned in section 2, a ner algorithm result may include a confidence score for each token for each of the possible classes. in this case the ner algorithm result (and predictor input) is a function f : t @xmath5. here, we propose the following predictors. ## datasets there is an apparent lack of arabic corpora (abouenour et al., 2010). moreover, few of these corpora have been made freely and publicly available for research purposes @xcite . while others are available but under licenses @xcite thus, researchers often rely on private corpora, which precludes reproducing previous work and comparing its performance over a consistent benchmark. in this work we use three corpora, noorcorp (sajadi and minaei, 2017) based on a different historical book circa 800 ad, anercorp @xcite , a modern corpus which has become a standard in arabic ner works. anercorp is based upon web-documents written in modern arabic collected from 316 articles from newspapers such as bbc foot_0 and aljazeera foot_1 . ## experiments we now present a series of empirical evaluations examining the following questions. which predictors are the most correlated with our desired quality measures ( § 6.2)? does our dynamic ensemble approach outperform the baseline approaches of using a single ner algorithm and using static ensemble learning ( § 6.3)? what is the performance impact of training ner algorithms over modern arabic text versus historic arabic for a historic arabic ner task ( § 6.4)? we begin by introducing the experimental setup. ## conclusion and future work we have proposed a dynamic ensemble model for ner and demonstrated its efficacy over a historical arabic task. we have shown that training mlbased ner algorithms over modern arabic text negatively impacts their performance over historical text, even when the training is combined with historical sources. this result highlights the need for more tagged historical datasets, such as the bedaya corpus contributed by this work. in future work, we intend to enhance the dynamic ensemble model by exploring additional predictors and alternative ensemble learning methods. we further intend to explore different ways to utilize modern arabic corpora in historical text analysis tasks.