Performance Analysis of A rabic Pre-trained Models on Named Entity Recognition Task (2023)

Abstract: Named Entity Recognition (NER) is a crucial task within natural language processing (NLP) that entails the identification and classification of entities, such as person, organization and location. This study delves into NER specifically in the Arabic language, focusing on the Algerian dialect. While previous research in NER has primarily concentrated on Modern Standard Arabic (MSA), the advent of social media has prompted a need to address the variations found in different Arabic dialects. Moreover, given the notable achievements of Large-scale pre-trained models (PTMs) based on the BERT architecture, this paper aims to evaluate Arabic pre-trained models using an Algerian dataset that covers different domains and writing styles. Additionally, an error analysis is conducted to identify PTMs’ limitations, and an investigation is carried out to assess the performance of trained MSA models on the Algerian dialect. The experimental results and subsequent analysis shed light on the complexities of NER in Arabic, offering valuable insights for future research endeavors.

## introduction the expression named entities recognition (ner) has been used for the first time at the 6th edition of the message understanding conference (muc) in november 1995 @xcite . the task of ner consisted in using sgml markers to identify entities in texts (names of persons, organizations, or places), temporal expressions, and numerical expressions ("currency" or "percentages"). since then, ner has become a starting point and an important part of many applications in natural language processing @xcite , such as: information extraction (ie) (kumar and starly, 2022), information retrieval (ir) @xcite , semantic annotation (sa) @xcite , machine translation (mt) @xcite , question answering (qa) systems @xcite ), text summarization @xcite ) and text clustering @xcite . the process of ner can be done according to three main approaches @xcite ) @xcite , @xcite : the symbolic or linguistic (rule-based) approach, where the main idea is to use linguistic knowledge (internal or external clues), dictionaries and gazetteers of proper names to establish a list of knowledge rules (called regular expressions or finite state transducers @xcite ). however, the principal inconvenience of this approach is that the rule-generation process is fastidious and time-consuming. the machine learning (ml) approach, is mainly based on a previously annotated corpus. where the recognition problem is converted into a classification problem and employs various ml models to solve it. a hybrid approach which combines the two previous approaches to boost the performance of the models developed have been tried as well. in recent years, the deep learning approach has proven to be a very powerful for learning feature representations directly from datasets, achieving outstanding results. the approach can learn complex hidden representations without complex feature engineering and rich domain knowledge @xcite . while the task for latin scripted language is more advanced @xcite , having features like capitalization gives a clue and differentiates between named entities and other words. such feature is absent in languages like arabic. the additional complexity of the task comes from the dialectal variations of arabic. in the literature, most of the works on ner in arabic have been oriented towards the common version msa, a variant that is both normalized and standardized. however, with the emergence of so-cial media (facebook, tweeter, youtube,. . . etc.) as a means of communication and also as a source of information. a huge amount of raw data generated every day, which represents a goldmine for many applications in nlp. therefore, the research on ner has been oriented towards these variants of the arabic language. dialectal arabic is another form of arabic language used in everyday' communications, and is generally spoken and written (social networks, advertisements, sms, etc.). it varies not only from one arab country to another, but also from one region to another within the same country. thus, almost all arab countries have their own dialects. arabic dialectology generally distinguishes two main areas or families of dialects @xcite , @xcite , @xcite ): various other granular classification were proposed in literature classify the dialects into five or more variants, namely gulf, nile basin, levant and maghreb @xcite @xcite to even city level @xcite . the algerian dialect, also known as darija ("common language"), is spoken by 70% to 80% of the algerian population @xcite (of estimated 45 million people). when we speak about algerian dialect, we must understand that it is a question of various sub-varieties of local dialect due to the geographical expansion of the country (2.382 million km²), because there is no unified algerian dialect. there are therefore many varieties of algerian dialect. it should be remembered that all these sub-varieties are heterogeneously influenced by other languages (e.g. berber, french, spanish, turkish, italian, etc.) @xcite . thus, we can distinguish algiers dialect (mainly influenced by berber and turkish), oranais dialect (influenced by spanish), constantinois dialect (influenced by italian), tlemc ¸ani dialect (influenced by andalusian arabic), etc. in the context of nlp, the algerian dialect constitutes a real challenge due to the multitude of constraints it presents, which are either inherited from standard arabic, such as agglutination, and syntactic flexibility. or they are due to the dialect itself, such as lack of normalization and standardization (it is common in algerian dialect as the case of other dialects to find several orthographic transcriptions for the same), code-switching (a consequence of alternating two or more languages (or varieties of dialect) during the production of the same sentences or conversation). arabizi is a new spontaneous spelling variant of algerian dialect, based particularly on latin characters associated with numbers of special characters. such challenges motivated us to explore and focus more on this dialect in an attempt to investigate its particularities in the context of the new deep learning models. our contributions in this paper can be summarized as follows: the remainder of this paper is organized as follows: the related work for ner in arabic is presented in section 2. section 3 gives some indications about arabic pre-trained models. section 4 and 5 are devoted to experiments and results. the error analysis is described in section 6 and finally, conclusion and future works are presented in section 7. ## related work the first work on aner was the tagarab system in 1998 @xcite . since then, many studies have followed covering different approaches: rule-based, machine learning or hybrid. in this section, we will divide the works into two categories: those on the algerian dialect which are rare and the second category is the works on msa adopting a deep learning approach. ## arabic pre-trained models pre-trained language models, including bert @xcite and roberta @xcite , have demonstrated significant success across a wide range of nlp tasks in various languages. arabic nlp has witnessed substantial advancements with the development of dedicated pre-trained language models, achieving state-of-the-art outcomes in both msa and da as shown in ## experimental setup this section details the experimental setup used in our research. in our experiments, we investigated the performance and limitations of the arabic pretrained model in the ner task. ## results and discussion we carried out a battery of experiments in the following order: ## error analysis for further investigation, we selected the best performing model arabert to probe and examine the shortfall of such class of models. for such task, we inspected the errors on dzner. figure 3 shows the confusion matrix for the results of evaluating dzner on model finetuned with the training set from the same dataset. it is clear that the majority of the errors are caused by not detecting pers, org and loc respectively on the order of error severity. looking deeper into the issue, we selected 100 samples among the errors resulted from the classification. we noted that the bulk of these errors are caused by lack of spelling standards such as the case of " " which are misspellings for " " respectively. such cases represents over 13% of the errors. while another large set of errors are caused by transliteration, this is mostly when using foreign or entities in another language but transcribing them in arabic. cases such as " " that represents " google, alger, algerie " respectively. such category of errors represent another 21% among the errors. errors such missing capitalization in latin transcribed entities is very common as well. this is the case for "bougara, paris, and zanzibar" that supposed to be transcribed with capitals as " bougara, paris, zanzibar ". such issues highlight the challenges dealing with dialectal content that is present in this dataset and similar ones. ## conclusion in this study, we conducted a series of experiments to investigate ner performance in the context of arabic, with a specific focus on the algerian dialect. our findings shed light on the challenges and limitations of existing arabic pre-trained models trained on msa and da when applied to dialectal content. the experiments comparing the performance of anercorp on dzner and vice versa revealed the difficulties posed by the lack of standardized orthography and extensive code-switching in dialectal content. while the fine tuned models showed slightly improved results on the msa dataset, the dialectal content encompassed msa features, highlighting the dominance of dialectal data in this task. the combination of the aner-corp and dzner datasets did not significantly impact the evaluation results, indicating that the datasets offer non-redundant features and are disjoint from each other. the error analysis, conducted using the best performing model arabert, identified common sources of errors in dialectal content, such as spelling variations, transliteration issues, and missing capitalization in latin transcribed entities. these findings emphasize the challenges associated with dialectal content and the need to address spelling variations and non-standardized writing in dialectal arabic. future research will focus on: (i) refining ner models to better handle dialectal arabic; (ii) explore strategies to expand these resources and improve performance in dialectal contexts; and (iii) investigate joint training ner with other auxiliary tasks such as part of speech tagging. both tasks can mutually benefit from each other and share useful knowledge.