BERT ifying the Hidden M arkov Model for Multi-Source Weakly Supervised Named Entity Recognition (2021)

Abstract: We study the problem of learning a named entity recognition (NER) tagger using noisy labels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, inaccurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hidden Markov model (CHMM), which can effectively infer true labels from multi-source noisy labels in an unsupervised way. CHMM enhances the classic hidden Markov model with the contextual representation power of pre-trained language models. Specifically, CHMM learns token-wise transition and emission probabilities from the BERT embeddings of the input tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM-ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT-NER’s output is regarded as an additional weak source to train the CHMM in return. Experiments on four NER benchmarks from various domains show that our method outperforms state-of-the-art weakly supervised NER models by wide margins.

## introduction named entity recognition (ner), which aims to identify named entities from unstructured text, is an information extraction task fundamental to many downstream applications such as event detection @xcite , relationship extraction @xcite , and question answering @xcite . existing ner models are typically supervised by a large number of training sequences, each pre-annotated with token-level labels. in practice, however, obtaining such labels could be prohibitively expensive. on the other hand, many domains have various knowledge resources such as knowledge bases, domain-specific dictionaries, or labeling rules provided by domain experts @xcite . these resources can be used to match a corpus and quickly create large-scale noisy training data for ner from multiple views. learning an ner model from multiple weak supervision sources is a challenging problem. while there are works on distantly supervised ner that use only knowledge bases as weak supervision @xcite @xcite , they cannot leverage complementary information from multiple annotation sources. to handle multi-source weak supervision, several recent works @xcite @xcite leverage the hidden markov model (hmm), by modeling true labels as hidden variables and inferring them from the observed noisy labels through unsupervised learning. though principled, these models fall short in capturing token semantics and context information, as they either model input tokens as one-hot observations @xcite or do not model them at all @xcite . moreover, the flexibility of hmm is limited as its transitions and emissions remain constant over time steps, whereas in practice they should depend on the input words. we propose the conditional hidden markov model (chmm) to infer true ner labels from multi-source weak annotations. chmm conditions the hmm training and inference on bert by predicting token-wise transition and emission probabilities from the bert embeddings. these tokenwise probabilities are more flexible than hmm's constant counterpart in modeling how the true labels should evolve according to the input tokens. the context representation ability they inherit from bert also relieves the markov constraint and expands hmm's context-awareness. further, we integrate chmm with a supervised bert-based ner mode with an alternate-training method (chmm-alt). it fine-tunes bert-ner with the denoised labels generated by chmm. taking advantage of the pre-trained knowledge contained in bert, this process aims to refine the denoised labels by discovering the entity patterns neglected by all of the weak sources. the finetuned bert-ner serves as an additional supervision source, whose output is combined with other weak labels for the next round of chmm training. chmm-alt trains chmm and bert-ner alternately until the result is optimized. our contributions include: the code and data used in this work are available at github.com/yinghao-li/chmm-alt. ## related work weakly supervised ner there have been works that train ner models with different weak supervision approaches. distant supervision, a specific type of weak supervision, generates training labels from knowledge bases @xcite @xcite @xcite . but such a method is limited to one source and falls short of acquiring supplementary annotations from other available resources. other works adopt multiple additional labeling sources, such as heuristic functions that depend on lexical features, word patterns, or document information @xcite , and unify their results through multi-source label denoising. several multi-source weakly supervised learning approaches are designed for sentence classification @xcite @xcite . although these methods can be adapted for sequence labeling tasks such as ner, they tend to overlook the internal dependency relationship between token-level labels during the inference. @xcite target the ner task, but their method first generates candidate named entity spans and then classifies each span independently. this independence makes it suffer from the same drawback as sentence classification models. a few works consider label dependency while dealing with multiple supervision sources. @xcite train a bilstm-crf network @xcite with multiple parallel crf layers, each for an individual labeling source, and aggregate their transitions with confidence scores predicted by an attention network @xcite . hmm is a more principled model for multi-source sequential label denoising as the true labels are implicitly inferred through unsupervised learning without deliberately assigning any additional scores. following this track, @xcite and @xcite use a standard hmm with multiple observed variables, each from one labeling source. @xcite propose linked hmm, which differs from ordinary hmm by introducing unique linking rules as an adjunct supervision source additional to general token labels. however, these methods fail to utilize the context information embedded in the tokens as effectively as chmm, and their ner performance is further constrained by the markov assumption. neuralizing the hidden markov model some works attempt to neuralize hmm in order to relax the markov assumption while maintaining its generative property @xcite . for example, @xcite and @xcite incorporate recurrent units into the hidden semi-markov model (hsmm) to segment and label highdimensional time series; @xcite learn discrete template structures for conditional text generation using neuralized hsmm. @xcite and @xcite factorize hmm with neural networks to scale it and improve its sequence modeling capacity. the work most related to ours leverages neural hmm for sequence labeling @xcite . chmm differs from neural hmm in that the tokens are treated as a dependency term in chmm instead of the observation in neural hmm. besides, chmm is trained with generalized em, whereas neural hmm opti- center in new york was... b-per source 1 o o b-loc i-loc o... b-loc source 2 i-loc o o b-loc o... b-loc target i-loc o i-loc o... b-loc rockefeller figure 1: an example of label aggregation with two weak labeling sources. we use bio labeling scheme. per represents person; loc is location. ## problem setup in this section, we formulate the multi-source weakly supervised ner problem. consider an input sentence that contains t tokens w ( foot_0 :t ) , ner can be formulated as a sequence labeling task that assigns a label to each token in the sentence. 1 assuming the set of target entity types is e and the tagging scheme is bio @xcite , ner models assign one label from the label set @xmath0, where the size of the label set is |l| = 2|e| + 1, e.g., if @xmath1, loc}, then @xmath2, b-per, i-per, b-loc, i-loc}. suppose we have a sequence with k weak sources, each of which can be a heuristic rule, knowledge base, or existing out-of-domain ner model. each source serves as a labeling function that generates token-level weak labels from the input corpus, as shown in figure 1 . for the input sequence w (1:t ) , we use x (1:t ) k , @xmath3, . . . , k} to represent the weak labels from the source k, where x (t) @xmath4, @xmath5, . . . , t } is a probability distribution over l. multi-source weakly supervised ner aims to find the underlying true sequence of labels ŷ(1:t ) , ŷ(t) ∈ l given {w (1:t ) , x (1:t ) 1:k }. ## methodology in this section, we describe our proposed method chmm-alt. we first sketch the alternate-training procedure ( § 4.1), then explain the chmm component ( § 4.2) and how bert-ner is involved ( § 4.3). ## experiments we benchmark chmm-alt on four datasets against state-of-the-art weakly supervised ner baselines, including both distant learning models and multi-source label aggregation models. we also conduct a series of ablation studies to evaluate the different components in chmm-alt's design. ## conclusion in this work, we present chmm-alt, a multisource weakly supervised approach that does not depend on manually labeled data to learn an accurate ner tagger. it integrates a label aggregator-chmm and a supervised model-bert-ner together into an alternate-training procedure. chmm conditions hmm on bert embeddings to achieve greater flexibility and stronger context-awareness. fine-tuned with chmm's prediction, bert-ner discovers patterns unobserved by the weak sources and complements chmm. training these models in turn, chmm-alt uses the knowledge encoded in both the weak sources and the pre-trained bert model to improve the final ner performance. in the future, we will consider imposing more constraints on the transition and emission probabilities, or manipulating them according to sophisticated domain knowledge. this technique could be also extended to other sequence labeling tasks such as semantic role labeling or event extraction.