GL a RA : Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognition (2021)

Abstract: Instead of using expensive manual annotations, researchers have proposed to train named entity recognition (NER) systems using heuristic labeling rules. However, devising labeling rules is challenging because it often requires a considerable amount of manual effort and domain expertise. To alleviate this problem, we propose GLARA, a graph-based labeling rule augmentation framework, to learn new labeling rules from unlabeled data. We first create a graph with nodes representing candidate rules extracted from unlabeled data. Then, we design a new graph neural network to augment labeling rules by exploring the semantic relations between rules. We finally apply the augmented rules on unlabeled data to generate weak labels and train a NER model using the weakly labeled data. We evaluate our method on three NER datasets and find that we can achieve an average improvement of +20% F1 score over the best baseline when given a small set of seed rules.

## introduction named entity recognition (ner) models often need to be trained with many manual labels to perform well. due to the high cost of manual annotations, collecting labeled data to train ner models is challenging for real-world applications. recently, researchers have proposed to collect weak labels using heuristic rules, which are called labeling rules @xcite @xcite . this kind of methods typically first ask domain experts to write labeling rules for a ner task, then use these manual rules to generate labeled data and train a ner model with the weakly labeled data. the advantage of these methods is that they do not require manual annotations. however, during our study, we find that writing labeling rules is also challenging for domain-specific tasks. devising accurate def rule1( x ): rule-1: associated_with_* → disease def rule2( x ): return "disease" if x.@xmath0 rule-2: cause_of_* → disease the symptoms were associated with enzyme deficiency. the fragile site is not associated with mental retardation. migraine is an uncommon cause of cranial neuropathy. the cause of hearing loss after spinal anaesthesia is unknown. rules often demands a significant amount of manual effort because it requires developers to have deep domain expertise and a thorough understanding of the target data. to alleviate manual effort on writing labeling rules, we propose glara, a graph-based labeling rule augmentation framework to automatically learn new rules from unlabeled data with a handful of seed rules. our work is motivated by the intuition that if two rules can accurately label the same type of entities, then they are semantically related via the entities matched by them. therefore, we can acquire new labeling rules based on their semantic relatedness with the rules that we have already known. for example, figure 1 shows two example rules for labeling disease entities. if we know that rule1 "associated with * →disease" is an accurate rule for labeling diseases, and rule1 is semantically related to rule2 "cause of * →disease", then we can derive that rule2 can be another accurate rule for labeling diseases. given unlabeled data and seeding rules, (1) we first extract candidate rules from unlabeled data, (2) then build a graph of rules and learn new rules by propagating the labeling information from seeding rules to other rules. (3) next we apply selected rules on unlabeled data and obtain a label matrix. (4) finally, we estimate the noisy labels using a generative model, and (5) train a final ner model with noisy labels. " * @xmath1. to augment labeling rules, we first define six types of rules and extract all possible rules from unlabeled data as candidate rules. then, for each rule type, we build a graph by connecting rules of this type based on their semantic similarities. in our work, we compute a rule's embedding as the average of contextual embeddings of entities matched by the rule, and compute the similarities between rules using their embeddings. we learn new rules using a graph neural network model from a small set of seed rules. next, we train a discriminative ner model using the weak labels generated by both the seeding and learned rules. to obtain weak training labels, we first obtain a label matrix by applying all the augmented rules on each token in unlabeled data. then, we estimate the labels of unlabeled instances using the linkedhmm model @xcite . we evaluate our framework on three datasets. in our experiments, we first show that our method can achieve better results than baselines when abundant rules are available. we also demonstrate that we can achieve an average improvement of +20% f1 when only a small set of rules are available. we summarize our major contributions as: ## the glara framework our goal is to build a ner system with a small set of manually selected seeding rules and unlabeled data. our key idea is to first augment labeling rules using graph neural networks, based on the hypothesis that semantically similar rules should have similar abilities to recognize entities. then, we train a ner model using the weak training data labeled by the augmented rules. overview figure 2 shows an example workflow of glara framework using suffix rules to recognize disease entities. our framework consists of five major components. (1) rule extractor: we define six types of rules and extract all possible rules from unlabeled data as candidate rules. (2) rule augmentation: for each rule type, we first build a graph of rules by connecting rules based on their semantic similarities. given a small set of manual seeding rules, we learn new rules by propagating the labeling confidence from seeding rules to other rules. (3) rule applier: we obtain a label matrix by applying all the augmented rules on each token in the unlabeled data. (4) generative model: we estimate the labels of unlabeled instances using a generative model. (5) discriminative model: we train a final discriminative ner model using the weak labels produced by the generative model. ## experimental setup in this section, we present the details of our experimental setup. first, we evaluate our method on three ner datasets, which we refer to ncbi, bc5cdr, and laptopreview. then, we describe the baseline methods compared in our experiments. we also give a detailed description of the seed rules used in our experiments. ## experimental results in this section, we first compare our method with state of the art methods using all available manual rules. then, we evaluate our method under scenarios when only limited seeding rules are available, which are very common in real-world applications. next, we conduct an ablation study to investigate the effectiveness of different types of rules and our newly proposed loss function based on centroid distance. finally, we also perform a quality analysis of the automatically learned labeling rules. ## related work training reliable ner systems usually requires large annotation efforts, which is often considered expensive and impractical in certain domains. therefore, previous studies have been trying to reduce the manual efforts required for annotation while producing comparable performance in ner tasks by utilizing manually designed rules that are cheap and accurate. studies have shown that human-defined rules (i.e dictionaries) can greatly aid ner tasks, especially in the domains where the identification of entities needs domain knowledge @xcite @xcite , and they can also be used to distantly create labeled data set to benefit machine learning models @xcite @xcite . while distantly labeled data sets can be created at a low cost to boost ner tasks, the models still suffer from the noise introduced by the imperfect rules. therefore, denoising models have been proposed to allow the model to better tolerate the im-perfect annotations created by rules. @xcite proposed autoner that trains ner systems using only lexicons with a tie-or-break tagging denoising model. similarly, some recent work @xcite @xcite have used a partial matching module to denoise the noisily labeled data sets. recently, weak supervision is proposed as another form of denoising framework without using any labeled data. weakly supervised systems use handcrafted labeling rules to create weak training instances from unlabeled data and then use a denoising generative model to approximate the posteriors of the rules. in this process, the unknown gold label is treated as latent variable by the generative model. the performance of the labeling rules could be further enhanced by training a neural network-based discriminative model by treating the posteriors as soft labels. related studies such as snorkel @xcite , swellshark @xcite , linkedhmm @xcite , and @xcite have demonstrated great success with carefully curated labeling rules. however, manually designing those high-quality rules often require domain expertise and easy to have a low sensitivity on identifying entities. in @xcite , the weak training data is created by broadly collecting available labeling rules from multiple sources, which demonstrates the importance of being able to automatically find new heuristics missed by human efforts. to find new heuristic rules on the basis of a relatively limited number of manually designed rules, previous studies have tried bootstrapping by relying on the co-occurrence, context and pattern features @xcite @xcite @xcite @xcite . recent studies on graph neural networks has opened up another possibility for learning new rules. by internally infusing the semantics of the neighboring nodes, the popular graph convolutional network (gcn) @xcite and graph attention network (gat) @xcite have shown great success in semisupervised node classification when the number of labeled nodes is limited. graph neural networks have been applied for many nlp tasks such as text classification @xcite @xcite , semantic role labeling (marcheggiani and titov, 2017), machine translation @xcite , question answering @xcite , information extraction @xcite @xcite @xcite @xcite , etc. in our work, we proposed to use graph neural networks to learn new labeling rules. based on graph attention network @xcite , we designed a new graph network model with a new loss function. experimental results demonstrated that our model performed better than the original graph attention network on learning accurate labeling rules. ## conclusion in this work, we proposed a weakly supervised ner framework that automatically learns highquality new rules from only a handful of manually designed rules with a graph-based labeling rule augmentation method (glara). experiments on three ner datasets demonstrate that our model outperforms baseline systems and achieves substantially better performance when the number of manual rules is limited. in addition, we also defined six types of rules that have been demonstrated useful for recognizing entities. in the future, we plan to improve glara by investigating more complex rule types and rule representation methods for weakly supervised ner.