Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss (2021)

Abstract: We study learning named entity recognizers in the presence of missing entity annotations. We approach this setting as tagging with latent variables and propose a novel loss, the Expected Entity Ratio, to learn models in the presence of systematically missing tags. We show that our approach is both theoretically sound and empirically useful. Experimentally, we find that it meets or exceeds performance of strong and state-of-the-art baselines across a variety of languages, annotation scenarios, and amounts of labeled data. In particular, we find that it significantly outperforms the previous state-of-the-art methods from Mayhew et al. (2019) and Li et al. (2021) by +12.7 and +2.3 F1 score in a challenging setting with only 1,000 biased annotations, averaged across 7 datasets. We also show that, when combined with our approach, a novel sparse annotation scheme outperforms exhaustive annotation for modest annotation budgets.1

## introduction named entity recognition (ner) is a critical subtask of many domain-specific natural language understanding tasks in nlp, such as information extraction, entity linking, semantic parsing, and question answering. for large, exhaustively annotated benchmark datasets, this problem has been largely solved by fine-tuning of high-capacity pretrained sentence encoders from massive-scale language modeling tasks @xcite @xcite . however, fully annotated datasets themselves are expensive to obtain at scale, creating a barrier to rapid development of models in low-resource situations. partial annotations, instead, may be much cheaper to obtain. for example, when building a dataset for a new entity extraction task, a domain expert may be able to annotate entity spans with high precision at a lower recall by scanning through documents inexhaustively, creating a higher diversity of contexts and surface forms by limiting the amount of time spent on individual documents. in another scenario studied by @xcite , non-speaker annotators for lowresource languages may only be able to recognize some of the more common entities in the target language, but will miss many less common ones. in both of these situations, we wish to leverage partially annotated training data with high precision but low recall for entity spans. because of the low recall, unannotated tokens are ambiguous and it is not reasonable to assume they are non-entities (the o tag). we give an example of this in figure 1 . we address the problem of training ner taggers with partially labeled, low-recall data by treating unannotated tags as latent variables for a discriminative tagging model. we propose to combine marginal tag likelihood training @xcite with a novel discriminative criterion, the expected entity ratio (eer), to control the relative proportion of entity tags in the sentence. the proposed loss is (1) flexibly able to incorporate prior knowledge about expected entity rates under uncertainty; (2) theoretically recovers the true tagging distribution under mild conditions; and (3) easy to implement, fast to compute, and amenable to standard gradient-based optimization. we evaluate our method across 7 corpora in 6 languages along two diverse low-recall annotation scenarios, one of which we introduce. we show that our method performs as well as or better than the previous state-of-the-art methods from @xcite and the recent work of @xcite across the studied languages, scenarios, and amounts of labeled entities. further, we show that our novel partial annotation scheme, when combined with our method, outperforms exhaustive annotation for modest annotation budgets. figure 1 : an example low-recall sentence with two entities (one is missing) and its ner tags. the gold row shows the true tags, the raw row shows a false negative induced by the standard ''tokens without entity annotations are non-entities'' assumption, and the latent row reflects our view of unannotated tags as latent variables. ## related works a common paradigm for low-recall ner is automatically creating silver-labeled data using outside resources. @xcite approach the problem by distantly supervising spans using a database with marginal tag training. @xcite similarly use a gazetteer and adapt the structured perceptron @xcite to handle partially labeled sequences, while @xcite use wikipedia and labelpropagation heuristics. @xcite also use distant supervision to silver-labeled entities, but use pu-learning with specified class priors to estimate individual classifiers with ad-hoc decoding. @xcite and @xcite optimize the marginal likelihood @xcite of the distantly annotated tags but require gazatteers and some fully labeled data to handle proper prediction of the o tag. @xcite use a marginal likelihood objective to pool overlapping ner tasks and datasets, but must exploit cross-dataset constraints. snorkel @xcite uses many sources of weak supervision, but relies on high recall and overlap to work. in contrast to these works, we do not use outside resources. our problem setting has connections to pulearning, which is classically an approach to classification @xcite @xcite , but here we work with tagging structures. our approach is also related to constraint-satisfaction methods for shaping the model distribution such as codl @xcite , used by @xcite , and is also related to posterior regularization @xcite , with main differences being that we do not use the kl-divergence and use gradient-based updates to a nonlinear model instead of closedform updates to a log-linear model. the problem setup from @xcite and @xcite is the same as ours, but @xcite use a cross-validated self-training approach and @xcite use an iterative constraint-driven self-training approach to down-weigh possible false-negative o tags, which they show to outperform @xcite . @xcite is the current state of the art on the conll 2003 ner datasets @xcite and we compare to their work in the experiments. recently, @xcite have published a span-based method that uses negative sampling of non-entity spans, but they do not provide any supporting theoretical guarantees. we also compare to them in the experiments. ## methods in this section, we describe the proposed approach. we begin with a description of the problem and notation in § 3.1, followed by the ner tagging model in § 3.2. we then describe the supervised marginal tag loss and our proposed auxiliary loss, used for learning on positive-only annotations, in § 3.3 and § 3.4, respectively. finally, in § 3.5 we describe the full objective and give theory showing that our approach recovers the true tagging distribution in the large-sample limit. ## benchmark experiments we evaluate our approach on 7 datasets in 6 languages for two diverse annotation scenarios (14 datasets in total) and compare to strong and stateof-the-art baselines. ## ee vs. exhaustive experiments in situations where we only have partially annotated data without the option for exhaustive annotations, the utility of being able to train with the data as provided is self-evident. however, given the potential upsides of partial annotation relative to exhaustive annotation-mentally less taxing and increased contextual diversity for a fixed annotation budget-it is natural to ask whether it is actually better to go with a sparse annotation scheme. ## conclusions we study learning ner taggers in the presence of partially labeled data and propose a simple, fast, and theoretically principled approach, the expected entity ratio loss, to deal with low-recall annotations. we show empirically that it outperforms the previous state of the art across a variety of languages, annotation scenarios, and amounts of labeled data. additionally, we give evidence that sparse annotations, when paired with our approach, are a viable alternative to exhaustive annotation for modest annotation budgets. though we study two simulated annotation scenarios to provide controlled experiments, our proposed eer approach is compatible with a variety of other incomplete annotation scenarios, such as incidental annotations (e.g., from web links on wikipedia), initialized by seed annotations from incomplete distant supervision/gazatteers, or embedded as a learning procedure in an active/iterative learning framework, which we intend to explore in future work.