A da K - NER : An Adaptive Top-K Approach for Named Entity Recognition with Incomplete Annotations (2022)

Abstract: State-of-the-art Named Entity Recognition (NER) models rely heavily on large amounts of fully annotated training data. However, accessible data are often incompletely annotated since the annotators usually lack comprehensive knowledge in the target domain. Normally the unannotated tokens are regarded as non-entities by default, while we underline that these tokens could either be non-entities or part of any entity. Here, we study NER modeling with incomplete annotated data where only a fraction of the named entities are labeled, and the unlabeled tokens are equivalently multi-labeled by every possible label. Taking multi-labeled tokens into account, the numerous possible paths can distract the training model from the gold path (ground truth label sequence), and thus hinders the learning ability. In this paper, we propose AdaK-NER, named the adaptive top-K approach, to help the model focus on a smaller feasible region where the gold path is more likely to be located. We demonstrate the superiority of our approach through extensive experiments on both English and Chinese datasets, averagely improving 2% in F-score on the CoNLL-2003 and over 10% on two Chinese datasets compared with the prior state-of-the-art works.

## introduction named entity recognition (ner) @xcite @xcite is a fundamental task in natural language processing (nlp). ner task aims at recognizing the meaningful entities occurring in the text, which can benefit various downstream tasks, such as question answering @xcite , event extraction @xcite , and opinion mining @xcite . strides in statistical models, such as conditional random field (crf) @xcite and pre-trained language models like bert @xcite , have equipped ner with new learning principles @xcite . pre-trained model with rich representation ability can discover hidden features automatically while crf can capture the dependencies between labels with the bio or bioes tagging scheme. however, most existing methods rely on large amounts of fully annotated information for training ner models @xcite . fulfilling such requirements is expensive and laborious in the industry. annotators, are not likely to be fully equipped with comprehensive domain knowledge, only annotate the named entities they recognize and let the others off, resulting in incomplete annotations. they typically do not specify the non-entity @xcite , so that the recognized entities are the only available annotations. figure 1( @xcite shows examples of both gold path foot_0 and incomplete path. for corpus with incomplete annotations, each unannotated token can either be part of an entity or non-entity, making the token equivalently multi-labeled by every possible label. since conventional crf algorithms require fully annotated sentences, a strand of literature suggests assigning weights to possible labels @xcite . fuzzy crf @xcite focused on filling the unannotated tokens by assigning equal probability to every possible path. further, @xcite introduced a weighted crf method to seek a more reasonable distribution q for all possible paths, attempting to pay more attention to those paths with high potential to be gold path. ideally, through comprehensive learning on q distribution, the gold path can be correctly discovered. however, this perfect situation is difficult to achieve in practical applications. intuitively, taking all possible paths into consideration will distract the model from the gold path, as the feasible region (the set of possible paths where we search for the gold path) grows exponentially with the length of the unannotated tokens increasing, which might cause failure to identify the gold path. to address this issue, one promising direction is to prune the size of feasible region during training. we assume the unknown gold path is among or very close to the top-k paths with the highest possibilities. specifically, we utilize a novel adaptive k-best loss to help the training model focus on a smaller feasible region where the gold path is likely to be located. furthermore, once a path is identified as a disqualified sequence, it will be removed from the feasible region. this operation can thus drastically eliminate redundancy without undermining the effectiveness. for this purpose, a candidate mask is built to filter out the less likely paths, so as to restrict the size of the feasible region. trained in this way, our adak-ner overcomes the shortcomings of fuzzy crf and weighted crf, resulting in a significant improvement on both precision and recall, and a higher f 1 score as well. in summary, the contributions of this work are: under the incomplete annotation framework, we introduce the following terminologies. a possible path refers to a possible complete label sequence consistent with the annotated tokens. for example, a possible incomplete annotated label sequence for x can be y @xmath0, y 2 , -, • • • , -), where token x 2 is annotated as y 2 and other missing labels are labeled as -. y @xmath1, y 2 , • • • , y cn ) with y @xmath2, is a possible path for x, where all the missing labelsare replaced by some elements in y . set c(y u ) denotes all possible complete paths for x with incomplete annotation y u . @xmath3, y i u )} is the available incompletely annotated dataset. for ner task, crf @xcite is a traditional approach to capture the dependencies between the labels by modeling the conditional probability p w (y|x) of a label sequence y given an input sequence x of length n as: φ(x, y) denotes the map from a pair of x and y to an arbitrary feature vector, w is the model parameter, p w (y|x) is the probability of y predicted by the model. once w has been estimated via minimizing negative log-likelihood: the original crf learning algorithm requires a fully annotated sequence y, thus the incompletely annotated data is not directly applicable to it. @xcite modified the loss function as follows: nonetheless, as figure 1 (b) shows, a sentence with only 6 words (1 annotated, 5 unannotated) have 3125 possible paths. we argued that identifying the gold path from all possible paths is like looking for a needle in a haystack. this motivates us to reduce redundant paths during training. we propose two major strategies (adaptive k-best loss and candidate mask) to induce the model to focus on the gold path (figure 1(c )), and two minor strategies (annealing technique and iterative sample selection) to further improve the model effectiveness in ner task. the workflow is summarized in algorithm 1. ## related works pre-trained language models has been an emerging direction in nlp since google launched bert @xcite in 2018. with the powerful transformer architecture, several pre-trained models, such as bert and generative pretraining model (gpt), and their variants have achieved stateof-the-art performance in various nlp tasks including ner @xcite . yang [2019] proposed a pre-trained permutation language model (xlnet) to overcome the limitations of denoising autoencoding based pretraining. @xcite demonstrated that more data and more parameter tuning could benefit pre-trained language models, and released a new pre-trained model (roberta). to follow the trend, we use bert as our neural model in this work. statistical modeling has been widely employed in sequence labeling. classical models learn label sequences through graph-based representation, with prominent examples such as hidden markov model (hmm), maximum entropy markov models (memm) and conditional random fields (crf) @xcite . among them, crf is an optimal model, since it resolves the labeling bias issue in memm and doesn't require the unreasonable independence assumptions in hmm. however, conventional crf is not directly applicable to the incomplete annotation situation. ni @xcite select the sentences with the highest confidence, and regarding missing labels as o. another line of work is to replace crf with partial crf @xcite or fuzzy crf @xcite , which assign unlabeled words with all possible labels and maximize the total probability @xcite . although these works have led to many promising results, they still need external knowledge for high-quality performance. @xcite presented a weighted crf model which is most closely related to our work. they estimated a proper distribution for all possible paths derived from the incomplete annotations. our work enhances fuzzy crf by reducing the possible paths by a large margin, aiming to better focus on the gold path. ## conclusion in this paper, we explore how to build an effective ner model by only using incomplete annotations. we propose two major strategies including introducing a novel adaptive k-best loss and a mask based on k-best candidates and self-built candidates to help our model better focus on the gold path. the results show that our approaches can significantly improve the performance of ner model with incomplete annotations.