Towards Realistic Single-Task Continuous Learning Research for NER (2021)

Abstract: There is an increasing interest in continuous learning (CL), as data privacy is becoming a priority for real-world machine learning applications. Meanwhile, there is still a lack of academic NLP benchmarks that are applicable for realistic CL settings, which is a major challenge for the advancement of the field. In this paper we discuss some of the unrealistic data characteristics of public datasets, study the challenges of realistic single-task continuous learning as well as the effectiveness of data rehearsal as a way to mitigate accuracy loss. We construct a CL NER dataset from an existing publicly available dataset and release it along with the code to the research community.

## introduction data privacy is a hot topic in ml, gaining attention in both industry and academia @xcite . one of the topics of interest is data retention, which can be improved by training models incrementally @xcite ). an ideal training regime would involve continuously updating a model on newly acquired data, then deleting the data. benchmarking cl strategies today is still highly nonstandard in academic research @xcite . one key difference between real-world and academic datasets is the dynamic nature of the former. academic datasets are often static and contain data that is annotated all at once based on fixed annotation guidelines. when building real-world applications, such data collection and annotation workflow is often not realistic. rather, an initial dataset is created and then is evolved over time based on usage pattern changes and business needs. for example, new labels are added periodically, * work completed while first author was an intern at amazon alexa ai. consequently, the academic cl research focus has been mainly on the multi-task learning scenario, where the same model is required to learn a number of isolated tasks incrementally without forgetting how to solve the previous ones. in this work we tackle the single-task scenario using the named entity recognition (ner) task. there is only one task, but it evolves over time due to data distribution shift, introduction of new labels, or other factors. single-task is often considered to be more difficult than multi-task @xcite @xcite and is also a common real-world scenario. to the best of our knowledge, there are no public nlp benchmarks specifically designed for singletask cl. in order to study this problem we pick the recent stackoverflowner dataset @xcite . the dataset authors' motivation was studying named entity recognition in the social computer programming domain, not continuous learning. however, the characteristics of the dataset are ideal for a study in cl. it spans roughly 10 years (from september 2008 to march 2018) of questionanswer threads that are manually annotated with close to 30 types of entities. the dataset is also very diverse and has a large sample size -other public ner datasets are too small or contain only a few entity types. finally, the manual annotation process resembles that of industrial use cases, where the labeling process might be subject to noise and human error. in order to simulate cl we split the data into time-based episodes and train an ner model incrementally over 5 episodes. our results show no regression and limited forgetting. to present a more realistic challenge, we propose a configurable r s i o n count ep 1 ep 2 ep 3 ep 4 ep 5 (a) temporal 0 200 400 600 800 1000 1200 1400 1600 1800 2000 a p p l i c a t i o distribution-based sampling of data inspired by our experiences with a confidential industrial dataset. we show that our sampled episodes can be used to study the effectiveness of different single-task cl strategies in the context of ner. the resulting dataset is the main contribution of this work. our cl model. our model design is inspired by lamol @xcite and adapted for ner. we employ a pre-trained gpt-2 language model base @xcite , then 2 layers of bi-lstm with 768 dimensions in each direction, a tanh non-linearity and linear transformation (1536 by number of labels), and a crf layer to predict labels. all parameters besides the gpt-2 base (pretrained on openai's webtext) are randomly initialized, and we train or finetune all parameters during training. training on all 5 episodes takes less than 12 hours on an nvidia tesla m40 gpu for all experimental settings. we assume that all entity types are known in advance so we do not need to expand the label size in a later episode if a new label is introduced. in our experiments, our baseline is a model fine-tuned on all training data. we compare the baseline to gdumb and two cl strategies: training with and without data replay. ## conclusions we demonstrate that even in an academic dataset spanning a decade, some important characteristics of applied single-task continual learning settings, such as data shift and label imbalance, are missing. we modify and release a dataset that contains some of these realistic challenges, and we establish a data replay baseline. although the ability to access and publish statistics for real industrial datasets is limited due to privacy and business concerns, we find that our dataset exhibits many important similarities to such datasets. our method for producing the dataset is configurable and can be used to build different degrees of data variance to support different use cases. although our dataset is a useful first step towards more realistic single-task continual learning, this work highlights the need for a public benchmark with truly continuous annotation.