BERT - P ers NER : A New Model for P ersian Named Entity Recognition (2021)

Abstract: Named entity recognition (NER) is one of the major tasks in natural language processing. A named entity is often a word or expression that bears a valuable piece of information, which can be effectively employed by some major NLP tasks such as machine translation, question answering, and text summarization. In this paper, we introduce a new model called BERT-PersNER (BERT based Persian Named Entity Recognizer), in which we have applied transfer learning and active learning approaches to NER in Persian, which is regarded as a low-resource language. Like many others, we have used Conditional Random Field for tag decoding in our proposed architecture. BERT-PersNER has outperformed two available studies in Persian NER, in most cases of our experiments using the supervised learning approach on two Persian datasets called Arman and Peyma. Besides, as the very first effort to try active learning in the Persian NER, using only 30% of Arman and 20% of Peyma, we respectively achieved 92.15%, and 92.41% performance of the mentioned supervised learning experiments.

## introduction named entity recognition (ner) is a fundamental natural language processing (nlp) task, in which we try to identify and classify certain entities such as organizations, persons, locations, etc. in a given text. a named entity is often a word or expression in the text that bears a valuable piece of information, which can be effectively used in other high-level nlp tasks such as machine translation, question answering, and text summarization. unlike english, which is rich in digital resources, there are some natural languages such as persian, which is regarded to be a low-resource language. however, persian is an important language because it is spoken by more than 110 million people worldwide. in this article, we introduce an efficient model for persian ner. to tackle problems such as the lack of labeled data in persian, we have used two different learning methods, i.e., transfer learning @xcite and active learning @xcite . to transfer knowledge in transfer learning, the modelbased (parameter) approach @xcite has been used to fine-tune bert @xcite . through transfer learning, we can produce a powerful model using limited data that takes much less training time. we also employed active learning in order to perform the fine-tuning task by using only a few informative samples instead of the whole dataset. through active learning, it is possible to deliver a performance that is very close to that of a supervised learning method. the structure of this paper is as follows. we review the literature in sec. 2; bert is discussed in sec. 3; our proposed method is explained in sec. 4; the results of our experiments are given in sec. 5 and the article is concluded in sec. 6. ## bert as it was mentioned before, the architecture of bert is based on transformer. bert is a pre-trained model, trained on about 3.3 billion unlabeled data. its input embedding consists of three parts: 1) token embedding, 2) segment embedding, and 3) position embedding @xcite . for token embedding, wordpiece has been employed. segment embedding is used to distinguish a pair of input sentences. the goal of position embedding is to determine sentence word order as the words of each sentence are fed to bert simultaneously and without any specific order @xcite . bert is applicable in both freezing and fine-tuning methods (pan and yang, 2010). in freezing, no change is applied to the pretrained model. freezing leads to the extraction of constant features from the model so that they can be used as contextualized word embeddings. in contrast, in the fine-tuning method, the parameters of certain layers are fine-tuned based on a down-stream task and labeled data. it was shown that fine-tuning outperforms freezing @xcite . therefore, we have used fine-tuning in our proposed method. besides, like several previous studies, we have used crf for the tag decoding. the detailed account of our proposed method, which we are going to refer to as bert-persner, is given next. ## conclusion we fine-tuned the pre-trained bert model for the task of named entity recognition in persian, which is regarded as a low-resource language. we employed both transfer learning and active learning methods to develop a new model called bert-persner for persian ner. the new model was evaluated on two persian ner datasets, which are called arman and peyma. we first evaluated bert-persner in a supervised approach, which was done on both word-and phrase-level. our new model outperformed two previous studies in persian ner by 2.05%, 2.73%, and 4.01% f1 scores, in phrase-level on peyma, in wordlevel on arman, and in phrase-level on arman, respectively. in the word-level evaluation on peyma, however, the performance of bert-persner was lower than the best available related work by 0.86% f1 score. bert-persner was evaluated word-level using the active learning approach with four different selection strategies. as our main contribution, it was shown that using only 30% of arman, we can achieve 92.15% performance of the supervised learning method. it was also shown that using only 20% of peyma, we can reach 92.41% performance of the supervised learning case. to the best of our knowledge, the application of active learning in persian ner is the very first effort in this respect. as our future work, we intend to investigate the impact of other selection strategies on bert-persner. we also plan to evaluate the pro-posed approach using other newly published pre-trained models.