T 2 NER : Transformers based Transfer Learning Framework for Named Entity Recognition (2021)

Abstract: Recent advances in deep transformer models have achieved state-of-the-art in several natural language processing (NLP) tasks, whereas named entity recognition (NER) has traditionally benefited from long-short term memory (LSTM) networks. In this work, we present a Transformers based Transfer Learning framework for Named Entity Recognition (T2NER) created in PyTorch for the task of NER with deep transformer models. The framework is built upon the Transformers library as the core modeling engine and supports several transfer learning scenarios from sequential transfer to domain adaptation, multi-task learning, and semi-supervised learning. It aims to bridge the gap between the algorithmic advances in these areas by combining them with the state-of-the-art in transformer models to provide a unified platform that is readily extensible and can be used for both the transfer learning research in NER, and for real-world applications. The framework is available at: https://github.com/suamin/t2ner .

## introduction named entity recognition (ner) is an important task in information extraction, benefiting the downstream applications such as entity linking @xcite , relation extraction @xcite and question answering @xcite . ner has been a challenging task in nlp due to large variations in entity names and flexibility in how entities are mentioned. these challenges are further enhanced in crosslingual and cross-domain ner settings, where the added difficulty comes from the difference in text genre and entity names across languages and domains @xcite . furthermore, ner models have shown relatively high variance even when trained on the same data @xcite . these models generalize poorly when tested on data from different domains and languages, and even more so when they contain unseen entity mentions @xcite @xcite . these challenges make transfer learning research an important and well studied area in ner. recent successes in transfer learning have mainly come from pre-trained language models @xcite with contextualized word embeddings based on deep transformer models @xcite . these models achieve state-of-the-art in several nlp tasks such as named entity recognition, document classification, and question answering. due to their wide success and the community adoption, successful frameworks like transformers have emerged. in ner, the existing frameworks like ncrf++ @xcite lack the core infrastructure to support such models directly with state-of-the-art transfer learning algorithms. in this paper, we present an adaptable and userfriendly development framework for growing research in transfer learning with deep transformer models for ner, with underexplored areas such as semi-supervised learning. this is in contrast to the standard lstm based approaches which have largely and successfully dominated the ner research. our framework is aimed to bridge several gaps with core design principles that are discussed in next section. ## design principles t2ner is divided into several components as shown in figure 1 . the core design principle is to seamlessly integrate the transformers @xcite library as the backend for modeling, while extending it to support different transfer learning scenarios with a range of existing algorithms. transformers offer optimized implementations of several deep transformer models, including bert @xcite , gpt @xcite , roberta @xcite , and xlm (conneau and lample, 2019) among others, with multi-gpu, distributed, and mixed precision training. the second design principle is inspired by previous pre-trained models in the computer vision: dassl.pytorch @xcite foot_0 and trans-learn @xcite foot_1 that unify domain adaptation, domain generalization, and semisupervised learning, thus allowing easy benchmarking, fair comparisons, and reproducibility. t2ner is the unification of these major algorithmic approaches to bridge the gap between the algorithms and advance transfer learning research in ner. lastly, the cross-lingual and cross-domain research in ner has itself proposed several advances, including multi-task and joint learning @xcite @xcite @xcite , adversarial learn-ing @xcite , feature transfer @xcite @xcite , newer architectures @xcite , parameter sharing @xcite @xcite , parameter generation @xcite , mixture-of-experts @xcite , and usage of external resources @xcite . therefore, our final design principle aims to unify these researches and offer a framework to test them with deep transformer models, wherever such an algorithmic abstraction is possible, while exploring new paradigms. ## conclusion and future work in this work we presented a transformer based framework for transfer learning research in named entity recognition (ner). we laid out the design principles, detailed out the architecture, and presented the transfer scenarios and some of the representative algorithms. t2ner offers to bridge the gap between growing research in deep transformer models, ner transfer learning, and domain adaptation. t2ner has the potential to serve as a unified benchmark for existing and newer algorithms with state-of-the-art models. for future work, we consider the following: