Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data (2021)

Abstract: Weak supervision has shown promising results in many natural language processing tasks, such as Named Entity Recognition (NER). Existing work mainly focuses on learning deep NER models only with weak supervision, i.e., without any human annotation, and shows that by merely using weakly labeled data, one can achieve good performance, though still underperforms fully supervised NER with manually/strongly labeled data. In this paper, we consider a more practical scenario, where we have both a small amount of strongly labeled data and a large amount of weakly labeled data. Unfortunately, we observe that weakly labeled data does not necessarily improve, or even deteriorate the model performance (due to the extensive noise in the weak labels) when we train deep NER models over a simple or weighted combination of the strongly labeled and weakly labeled data. To address this issue, we propose a new multi-stage computational framework – NEEDLE with three essential ingredients: (1) weak label completion, (2) noise-aware loss function, and (3) final fine-tuning over the strongly labeled data. Through experiments on E-commerce query NER and Biomedical NER, we demonstrate that NEEDLE can effectively suppress the noise of the weak labels and outperforms existing methods. In particular, we achieve new SOTA F1-scores on 3 Biomedical NER datasets: BC5CDR-chem 93.74, BC5CDR-disease 90.69, NCBI-disease 92.28.

## introduction named entity recognition (ner) is the task of detecting mentions of real-world entities from text and classifying them into predefined types. for example, the task of e-commerce query ner is to identify the product types, brands, product attributes of a given query. traditional deep learning * work was done during internship at amazon. to achieve better performance with limited labeled data, researchers resort to large unlabeled data. for example, @xcite propose to pre-train the model using masked language modeling on large unlabeled open-domain data, which is usually hundreds/thousands of times larger than the manually/strongly labeled data. however, opendomain pre-trained models can only provide limited semantic and syntax information for domainspecific tasks. to further capture domain-specific information, @xcite ; @xcite propose to continually pre-train the model on large in-domain unlabeled data. when there is no labeled data, one approach is to use weak supervision to generate labels automatically from domain knowledge bases @xcite . for example, @xcite match spans of unlabeled biomedical documents to a biomedical dictionary to generate weakly labeled data. @xcite further show that by merely using weakly labeled data, one can achieve good performance in biomedical ner tasks, though still underperforms supervised ner models with manually labeled data. throughout the rest of the paper, we refer to the manually labeled data as strongly labeled data for notational convenience. while in practice, we often can access both a small amount of strongly labeled data and a large amount of weakly labeled data, generated from large scale unlabeled data and domain knowledge bases. a natural question arises here: "can we simultaneously leverage small strongly and large weakly labeled data to improve the model performance?" the answer is yes, but the prerequisite is that you can properly suppress the extensive labeling noise in the weak labels. the weak labels have three features: 1) "incompleteness": some entity mentions may not be assigned with weak labels due to the limited coverage of the knowledge base; 2) "labeling bias": some entity mentions may not be labeled with the correct types, and thus weak labels are often noisy; 3) "ultra-large scale": the weakly labeled data can be hundreds/thousands of times larger than the strongly labeled data. an ultra-large volume of weakly labeled data contains useful domain knowledge. but it also comes with enormous noise due to the "incompleteness" and "labeling bias" of weak labels. the enormous noise can dominate the signal in the strongly and weakly labeled data, especially when combined with the unsupervised pre-training techniques. such noise can be easily overfitted by the huge neural language models, and may even deteriorate the model performance. this is further corroborated by our empirical observation (see section 4) that when we train deep ner models over a simple or weighted combination of the strongly labeled and weakly labeled data, the model performance almost always becomes worse. to address such an issue, we propose a threestage computational framework named needle (noise-aware weakly supervised continual pretraining). at stage i, we adapt an open-domain pre-trained language model to the target domain by in-domain continual pre-training on the large in-domain unlabeled data. at stage ii, we use the knowledge bases to convert the in-domain unlabeled data to the weakly labeled data. we then conduct another continual pre-training over both the weakly and strongly labeled data, in conjunction with our proposed weak label completion procedure and noise-aware loss functions, which can effectively handle the"incompleteness" and "noisy labeling" of the weak labels. at stage iii, we finetune the model on the strongly labeled data again. the last fine-tuning stage is essential to the model fitting to the strongly labeled data. we summarize our key contributions as follows: ## preliminaries we briefly introduce the ner problem and the unsupervised language model pre-training. ## method to harness the power of weakly labeled data, we propose a new framework -needle, which contain stages as illustrated in figure 1 : 1) we first adapt an open-domain pre-trained language model to the downstream domain via mlm continual pre-training on the unlabeled in-domain data. ## experiments we use transformer-based open-domain pretrained models, e.g., bert, mbert, roberta-large, @xcite with a crf layer as our base ner models. throughout the experiments, we use the bio tagging scheme @xcite . for stages i and ii, we train the models for one epoch with batch size 144. for stage iii, we use the grid search to find optimal hyperparameters: we search the number of epochs in @xcite @xcite @xcite @xcite @xcite @xcite and batch size in @xcite @xcite . we use adam optimizer with a learning rate of 5 × 10 -5 on the e-commerce query ner dataset. for e-commerce query ner, we consider two settings: english queries and multilingual queries. for english ner, there are 10 different entity types, while the multilingual ner has 12 different types. the queries are collected from search queries to a shopping website. the unlabeled in-domain data and the weak annotation is obtained by aggregating user behavior data collected from the shopping website. we give more details about the weakly labeled data in appendix e. for biomedical ner, we use three popular benchmark datasets: bc5cdr-chem, bc5cdr-disease @xcite , and ncbi-disease @xcite . these datasets only contain a single entity type. we use the pre-processed data in bio format from @xcite following biobert @xcite and pubmedbert @xcite . we collect unlabeled data from pubmed 2019 baseline foot_0 , and use the dictionary lookup and exact string match to generate weak labels foot_1 . we only include sentences with at least one weak entity label. ## discussion and conclusion our work is closely related to fully weakly supervised ner. most of the previous works only focus on weak supervision without strongly labeled data @xcite @xcite . however, the gap between a fully weakly supervised model and a fully supervised model is usually huge. for example, a fully supervised model can outperform a weakly supervised model (au-toner, shang et al. ( 2018 )) with only 300 articles. such a huge gap makes fully weakly supervised ner not practical in real-world applications. our work is also relevant to semi-supervised learning, where the training data is only partially labeled. there have been many semisupervised learning methods, including the popular self-training methods used in our experiments for comparison @xcite @xcite @xcite @xcite . different from weak supervision, these semi-supervised learning methods usually has a partial set of labeled data. they rely on the labeled data to train a sufficiently accurate model. the unlabeled data are usually used for inducing certain regularization to further improve the generalization performance. existing semi-supervised learning methods such as self-training doesn't leverage the knowledge from weak supervision and can only marginally improve the performance. different from previous studies on fully weakly supervised ner, we identify an important research question on weak supervision: the weakly labeled data, when simply combined with the strongly labeled data during training, can degrade the model performance. to address this issue, we propose a new computational framework named needle, which effectively suppresses the extensive noise in the weak labeled data, and learns from both strongly labeled data and weakly labeled data. our proposed framework bridges the supervised ner and weakly supervised ner, and harnesses the power of weak supervision in a principled manner. note that, needle is complementary to fully weakly supervised / semi-supervised learning. one potential future direction is to combine nee-dle with other fully weakly supervised / semisupervised learning techniques to further improve the performance, e.g., contrastive regularization @xcite .