Evaluating Pretrained Transformer-based Models on the Task of Fine-Grained Named Entity Recognition (2020)

Abstract: Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task and has remained an active research field. In recent years, transformer models and more specifically the BERT model developed at Google revolutionised the field of NLP. While the performance of transformer-based approaches such as BERT has been studied for NER, there has not yet been a study for the fine-grained Named Entity Recognition (FG-NER) task. In this paper, we compare three transformer-based models (BERT, RoBERTa, and XLNet) to two non-transformer-based models (CRF and BiLSTM-CNN-CRF). Furthermore, we apply each model to a multitude of distinct domains. We find that transformer-based models incrementally outperform the studied non-transformer-based models in most domains with respect to the F1 score. Furthermore, we find that the choice of domains significantly influenced the performance regardless of the respective data size or the model chosen.

## introduction named entity recognition (ner) is part of the fundamental tasks in natural language processing (nlp). the main objective of ner is to detect and classify proper names (named entities) in a free text. typically, named entities can be subdivided into four broad categories: persons, i.e., first and last names, locations such as countries or landscapes, organisations such as companies or political parties, and miscellaneous entities which serves as a catch-all category for other named entities such as brands, meals, or social events. ner is an active research field and state-of-the-art solutions such as spacy foot_0 , flair @xcite , and primer foot_1 manage to achieve near-human performance. however, classical ner (which we refer to as coarse-grained ner in this paper) models typically distinguish between only a small number of entity types, usually fewer than a dozen distinct categories. while this kind of shallow classification is sufficient for many applications, there are industrial usecases in which more precise information is necessary such as financial documents processing in the banking and finance context. for instance, application forms for a business loan are usually supplied with several supporting textual documents. these can contain the names of different types of persons, such as the owner or the ceo of the applying company, the contact person(s) at the issuing bank, finance analysts, or lawyers. the same is true for organisation names such as the name of the issuing bank, a government agency, or the name of the applying company or third-party companies. it is necessary to not only detect entity names, but to also qualify and differentiate between various entity types. indeed, in many contexts the actual name of an entity is important only if it can be associated to a role, or any other relevant quality. in the banking and finance world for example, the strict regulatory requirements cannot be satisfied with just a list of who is involved; knowing how entities are involved is a necessity. the term "fine-grained named entity recognition" (fg-ner) was first coined by @xcite . it describes a subtask of ner, where the objective remains the same as standard ner, but where the number of entity types is considerably higher. in extreme cases, fg-ner models such as the fine-grained entity recognizer (figer) @xcite are able to distinguish between more than 100 distinct labels. conditional random field (crf) models @xcite have been popular for numerous sequence-to-sequence tasks such as ner. they perform reasonably well and can serve as a baseline for the task of fg-ner. in a previous study, @xcite compared the performance of several fg-ner approaches for the english and japanese languages. they found that the bilstm-cnn-crf model devised by @xcite combined with gazetteers performed the best in terms of f1 score for the english language. they also found that bilstm-cnn-crf performed well without the use of gazetteers. in fact, among the models that did not make use of gazetteers, bilstm-cnn-crf achieved the highest f1 score. in 2017, the introduction of the transformer model @xcite revolutionised the nlp landscape and led to a number of novel language modeling approaches which manage to outperform state-of-the-art models in numerous tasks. in 2018, @xcite developed the bidirectional encoder representations from transformers (bert) model, a powerful language modeling technique which is considered as one of the most significant breakthroughs in nlp in recent memory. bert models are pretrained on masked language modeling (mlm) and next sentence prediction (nsp) tasks. devlin et al. (2019) fine-tuned the resulting models on several fundamental nlp tasks such as the glue language understanding tasks @xcite , the squad question answering task @xcite , and the swag common sense inference task @xcite , for which bert manages to achieve state-of-the-art performances. furthermore, @xcite reported an f1 score of 92.8% when fine-tuned on the conll-2003 dataset for ner @xcite , achieving similar results as state-of-the-art models such as contextual string embeddings @xcite and elmo embeddings @xcite . improving on the bert model, @xcite at facebook ai foot_2 developed a robustly optimized bert approach (roberta). they claim that the standard bert models were undertrained and proposed a new version of bert that was trained for a longer time, on longer sequences, on more data, and with larger batches. furthermore, they trained only on the mlm task and with dynamic changes of the masking patterns applied to training data. bert's pretraining steps was performed on the same dataset using the same masked locations for the entire mlm task. roberta mitigated that problem by duplicating their dataset ten times, and using different masking patterns for each duplicate. they report that fine-tuned models derived from roberta either matched or improved on bert models in terms of performance, although they did not perform tests specifically on the ner task. we use the ewnertc dataset published by @xcite , containing roughly 7 million data samples in 49 different domains. to the best of our knowledge, our study is the first aiming to precisely evaluate the performance of these existing approaches on the fg-ner task. ## experimental setup in this section, we present the dataset used in this study and we introduce the different models that we compare against each other. ## experimental results in this section, we will answer the three research questions that we formulated for this study (cf. section 1). ## threats to validity this study was conducted on the ewnertc dataset @xcite which was annotated automatically. we are operating under the assumption that the annotations are accurate. however, while @xcite conducted an evaluation for the turkish counterpart of the dataset (twnertc), they did not evaluate the english one. nevertheless, ewnertc is the largest publicly available dataset that we could find and that is relevant for fg-ner studies. we further proposed to reduce the potential noise in labelling by considering only the subset associated to top labels (cf. section 2.1). performance measurements can be impacted by sub-optimal implementation of algorithms. to mitigate this threat, we collected the models' implementations that were released by their original authors, and already leveraged in previous studies, and we reused them in the settings they were designed for. while we conducted grid searches to determine optimised hyperparameters for the crf, bert, roberta and xlnet models, we did not specifically optimise the hyperparameters for the the bilstm-cnn-crf model due to the induced computational costs. furthermore, as pointed out in section 2, due to the large number of domains, we decided against using gazetteers even though they would likely have increased the f1-scores of the non-transformer-based models. ## conclusion in this paper, we presented an empirical study of the performance of various transformer-based models for the fg-ner task on a multitude of domains and compared them to both crf and bilstm-cnn-crf models (which are commonly used in the literature for the ner task). we concluded that while the transformer-based models did not manage to outperform non-transformerbased models in terms of precision, we observed a consistent increase in recall and f1 scores in most domains. we noticed, however, significant differences in performance for a selection of domains that could not be explained by the size of the respective datasets. this study yields the main insight that while transformer-based models can indeed bring significant performance improvements, they do not necessarily revolutionise the achievements in fg-ner to the same extent they did in other nlp tasks.