Subsequence Based Deep Active Learning for Named Entity Recognition (2021)

Abstract: Active Learning (AL) has been successfully applied to Deep Learning in order to drastically reduce the amount of data required to achieve high performance. Previous works have shown that lightweight architectures for Named Entity Recognition (NER) can achieve optimal performance with only 25% of the original training data. However, these methods do not exploit the sequential nature of language and the heterogeneity of uncertainty within each instance, requiring the labelling of whole sentences. Additionally, this standard method requires that the annotator has access to the full sentence when labelling. In this work, we overcome these limitations by allowing the AL algorithm to query subsequences within sentences, and propagate their labels to other sentences. We achieve highly efficient results on OntoNotes 5.0, only requiring 13% of the original training data, and CoNLL 2003, requiring only 27%. This is an improvement of 39% and 37% compared to querying full sentences.

## introduction the availability of large datasets has been key to the success of deep learning in natural language processing (nlp). this has galvanized the creation of larger datasets in order to train larger deep learning models. however, creating high quality datasets is expensive due to the sparsity of natural language, our inability to label it efficiently compared to other forms of data, and the amount of prior knowledge required to solve certain annotation tasks. such a problem has motivated the development of new active learning (al) strategies which aim to efficiently train models, by automatically identifying the best training examples from large amounts of unlabeled data @xcite @xcite . this tremendously reduces human annotation effort as much fewer instances need to be labeled manually. to minimise the amount of data needed to train a model, al algorithms iterate between training a model, and querying information rich instances to human annotators from a pool of unlabelled data @xcite . this has been shown to work well when the queries are 'atomic'-a single annotation requires a unit labour, and describes entirely the instance to be annotated. conversely, each instance of structured data, such as sequences, require multiple annotations. hence, such query selection methods can result in a waste of annotation budget @xcite . for example, in named entity recognition (ner), each sentence is usually considered an instance. however, because each token has a separate label, annotation budgeting is typically done on a token basis @xcite . budget wasting may therefore arise from the heterogeneity of uncertainty across each sentence; a sentence can contain multiple subsequences (of tokens) of which the model is certain on some and uncertain on others. by making the selection at a sentence level, although some budget is spent on annotating uncertain subsequences, the remaining budget may be wasted on annotating subsequences for which an annotation is not needed. it can therefore be desirable for annotators to label subsequences rather than the full sentences. this gives a greater flexibility to al strategies to locate information rich parts of the input with improved efficiency -and reduces the cognitive demands required of annotators. annotators may in fact perform better if they are asked to annotate shorter sequences, because longer sentences can cause boredom, fatigue, and inaccuracies @xcite . in this work, we aim to improve upon the efficiency of al for ner by querying for subsequences within each sentence, and propagating labels to unseen, identical subsequences in the dataset. this strategy simulates a setup in which annotators are presented with these subsequences, and do not have access to the full context, ensuring that their focus is centred on the tokens of interest. we show that al algorithms for ner tasks that use subsequences, allowing training on partially labelled sentences, are more efficient in terms of budget than those that only query full sentences. this improvement is furthered by generalising existing acquisition functions ( § 4.1) for use with sequential data. we test our approaches on two ner datasets, ontonotes 5.0 and conll 2003. on ontonotes 5.0, @xcite achieve stateof-the-art performance with 25% of the original dataset querying full sentences, while we require only 13% of the dataset querying subsequences. on conll 2003, we show that the al strategy of @xcite requires 50% of the dataset to achieve the same results as training on the full dataset, while ours requires only 27%. contributions of this paper are: ## related work al algorithms aim to query information rich data points to annotators in order to improve the performance of the model in a data efficient way. traditionally these algorithms choose data points which lie close to decision boundaries @xcite , where uncertainty is high, in order for the model to learn more useful information. this measure of uncertainty, measured through acquisition functions, are therefore vital to al. key functions include predictive entropy (maxent) @xcite , mutual information between model posterior and predictions (bald) @xcite , or the certainty of the model when making label predictions (here called lc) @xcite . these techniques ensure all instances used for training, painstakingly labelled by experts, have maximum impact on model performance. there has been exploration of uncertainty and deep learning based al for ner @xcite @xcite . these approaches however, treat each sentence as a single query instead of a collection of individually labelled tokens. in these methods, the acquisition functions that score sentences aggregate token-wise scores (through summation or averaging). other works forgo this aggregation, querying single tokens at a time @xcite @xcite . these works show that al for ner can be improved by taking the single token as a unit query, and use semi-supervision @xcite for training on partially labelled sentences @xcite . however, querying single-tokens is inapplicable in practise because, either a) annotators have access to the full sentence when queried but can only label one token, which would lead to frustration as they are asked to read the full sentence but only annotate a single token, or b) annotators only have access to the token of interest, which means that they would not have enough information to label tokens differently based on their context, leading to annotators labeling any unique token with the same label. moreover, if the latter approach was somehow possible, we would be able to reduce the annotation effort to the annotation of only the unique tokens forming the dataset, its dictionary. furthermore, all of these past works use conditional random fields (crfs) @xcite , which have since been surpassed as the state-of-the-art for ner (and most nlp tasks) by deep learning models @xcite . in this work we follow the approach where annotators only have access to subsequences of multiple tokens. however, instead of making use of single tokens, we will query more than one token, providing enough context to the annotators. this allows the propagation of these annotations to identical subsequences in the dataset, further reducing the total annotation effort. ## subsequence acquisition in this section we describe how we build on past works, and the core contribution of this paper. our work forms a more flexible al algorithm that operates on subsequences, as opposed to full sentences @xcite . this is achieved by generalising acquisition functions for subsequences ( § 4.1) scoring and querying subsequences within sentences ( § 4.2), and performing label propagation on unseen sentences to avoid the multiple annotations of repeated subsequences ( § 4.3). ## conclusion & future work in this study we have employed subsequence querying methods for improving the efficiency of al for ner tasks. we have seen that these methods outperform full sentence querying in terms of annotations required for optimal model performance, requiring 38.8% and 36.6% fewer tokens for the ontonotes 5.0 and conll 2003 datasets. optimal results for subsequence querying (and full sentence querying) were achieved by generalising previously used al acquisition functions, defining a larger family of acquisition functions for sequential data. the analysis of § 6.3 suggests that a full sentence querying causes noisy acquisition functions due to the tokens in the queried sentences that were not highly scored. this added noise reduces the budget efficiency, and a subsequence querying method eliminates a large part of this effect. this efficiency also translated into a faster recall of named entities in the dataset to be queried ( § 6.2). limitations and future work: limitations of this study are largely centred on the use of an oracle to provide tokens with their labels. with human annotators, the cropped context of subsequence queries may make them produce more inaccuracies than when annotating full sentences. such studies will help reveal how context affects label accuracy, how this, in turn, affects optimal hyperparameters in the subsequence selection process (such as optimal query length), further accommodations that must be made to effectively optimise worker efficiency, and how to deal with unreliable labels. we leave to future work the evaluation of these querying methods with human annotators. there are also ways to incorporate model generated labelling methods for more robust semi-supervision into our framework that we leave to future work. finally, there are examples of other tasks for structured data, such as audio, video, and image segmentation, where the part of an instance may be queried. a generalisation of the strategy demonstrated for the ner case may allow for more efficient active learning querying methods for these other types of data.