T - NER : An All-Round Python Library for Transformer-based Named Entity Recognition (2021)

Abstract: Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.

## introduction language model (lm) pretraining has become one of the most common strategies within the natural language processing (nlp) community to solve downstream tasks @xcite @xcite @xcite . lms trained over large textual data only need to be finetuned on downstream tasks to outperform most of the task-specific designed models. among the nlp tasks impacted by lm pretraining, named entity recognition (ner) is one of the most prevailing and practical applications. however, the availability of open-source ner libraries for lm training is limited. 3 in this paper, we introduce t-ner, an opensource python library for cross-domain analysis for ner with pretrained transformer-based lms. figure 1 shows a brief overview of our library and its functionalities. the library facilitates ner experimental design including easy-to-use features such as model training and evaluation. most notably, it enables to organize cross-domain analyses such as training a ner model and testing it on a different domain, with a small configuration. we also report initial experiment results, by which we show that although cross-domain ner is challenging, if it has an access to new domains, lm can successfully learn new domain knowledge. the results give us an insight that lm is capable to learn a variety of domain knowledge, but an ordinary finetuning scheme on single dataset most likely causes overfitting and results in poor domain generalization. as a system design, t-ner is implemented in pytorch @xcite on top of the transformers library @xcite . moreover, the interfaces of our training and evaluation modules are highly inspired by scikit-learn @xcite , enabling an interoperability with recent models as well as integrating them in an intuitive way. in addition to the versatility of our toolkit for ner experimentation, we also include an online demo and robust pre-trained models trained across domains. in the following sections, we provide a brief overview about ner in section 2, explain the system architecture of t-ner with a few basic usages in section 3 and describe experiment results on cross-domain transfer with our library in section 4. ## named entity recognition given an arbitrary text, the task of ner consists of detecting named entities and identifying their type. for example, given a sentence "dante was born in florence.", a ner model are would identify "dante" as a person and "florence" as a location. traditionally, ner systems have relied on a classification model on top of hand-engineered feature sets extracted from corpora @xcite , which was improved by carefully designed neural network approaches @xcite @xcite . this paradigm shift was mainly due to its efficient access to contextual information and flexibility, as human-crafted feature sets were no longer required. later, contextual representations produced by pretrained lms have improved the generalization abilities of neural network architectures in many nlp tasks, including ner @xcite . in particular, lms see millions of plain texts during pretraining, a knowledge that then can be leveraged in downstream nlp applications. this property has been studied in the recently literature by probing their generalization capacity @xcite @xcite . when it comes to lm generalization studies in ner, the literature is more limited and mainly restricted to indomain @xcite or multilingual settings @xcite . our library facilitates future research in cross-domain and cross-lingual generalization by providing a unified benchmark for several languages and domain as well as a straightforward implementation of ner lm finetuning. ## t-ner: an overview a key design goal was to create a self-contained universal system to train, evaluate, and utilize ner models in an easy way, not only for research purpose but also practical use cases in industry. moreover, we provide a demo web app (figure 2 ) where users can get predictions from a trained model given a sentence interactively. this way, users (even those without programming experience) can conduct qualitative analyses on their own or existing pre-trained models. in the following we provide details on the technicalities of the package provided, including details on how to train and evaluate any lm-based architecture. our package, t-ner, allows practitioners in nlp to get started working on ner with a few lines of code while diving into the recent progress in lm finetuning. we employ python as our core implementation, as is one of the most prevailing languages in the machine learning and nlp communities. our library enables python users to access its various kinds of features such as model training, in-and cross-domain model evaluation, and an interface to get predictions from trained models with minimum effort. ## evaluation in this section, we assess the reliability of t-ner with experiments in standard ner datasets. ## conclusion in this paper, we have presented a python library to get started with transformer-based ner models. this paper especially focuses on lm finetuning, and empirically shows the difficulty of crossdomain generalization in ner. our framework is designed to be as simple as possible so that any level of users can start running experiments on ner on any given dataset. to this end, we have also facilitated the evaluation by unifying some of the most popular ner datasets in the literature, including languages other than english. we believe that our initial experiment results emphasize the importance of ner generalization analysis, for which we hope that our open-source library can help nlp community to convey relevant research in an efficient and accessible way.