Weakly Supervised Named Entity Tagging with Learnable Logical Rules (2021)

Abstract: We study the problem of building entity tagging systems by using a few rules as weak supervision. Previous methods mostly focus on disambiguating entity types based on contexts and expert-provided rules, while assuming entity spans are given. In this work, we propose a novel method TALLOR that bootstraps high-quality logical rules to train a neural tagger in a fully automated manner. Specifically, we introduce compound rules that are composed from simple rules to increase the precision of boundary detection and generate more diverse pseudo labels. We further design a dynamic label selection strategy to ensure pseudo label quality and therefore avoid overfitting the neural tagger. Experiments on three datasets demonstrate that our method outperforms other weakly supervised methods and even rivals a state-of-the-art distantly supervised tagger with a lexicon of over 2,000 terms when starting from only 20 simple rules. Our method can serve as a tool for rapidly building taggers in emerging domains and tasks. Case studies show that learned rules can potentially explain the predicted entities.

## introduction entity tagging systems that follow supervised training, while accurate, often require a large amount of manual, domain-specific labels, making them difficult to apply to emerging domains and tasks. to reduce manual effort, previous works resort to manual lexicons @xcite or heuristic rules provided by domain experts @xcite @xcite as weak supervision. for example, linkedhmm @xcite can achieve performance close to supervised models using 186 heuristic rules in addition to a lexicon of over two million terms. however, it is challenging for experts to write complete and accurate rules or lexicons in emerging domains, which requires both a significant amount of manual effort and a deep understanding of the target data. how to build accurate entity tagging systems using less manual effort is still an open problem. in this work, we explore methods that can automatically learn new rules from unlabeled data and a small set of seed rules (e.g. 20 rules). such methods are desirable in real-world applications not only because they can be rapidly deployed to new domains or customized entity types, but also because the learned rules are often effective, interpretable, and simple for non-experts to "debug" incorrect predictions. as explained in figure 1 , new rules can be learned from seed rules. specifically, we propose a novel iterative learning method tallor that can learn accurate rules to train a neural tagger in an automated manner, with goal to address two key issues during learning process: (1) how to detect entity boundaries and predict their types simultaneously with rules, (2) how to generate accurate and diverse pseudo labels from rules. with such a small set of seed rules as supervision, previous works @xcite @xcite only focus on disambiguating entity types assuming entity spans are given or just syntactic chunks (e.g., noun phrases). however, we find that syntactic chunks often do not align well with target entity spans. for example, given a sentence from conll2003: "germany's representative to the european union's veterinary committee...", the noun phrases 1 are "germany's representative" and "the european union's veterinary committee", but gold entities in the sentence are "germany" and "european union". we used noun phrases extracted from spacy as predicted entity boundaries and compared them with ground truth entity boundaries, which are extracted based on the results from syntactic parsing. this setting of using noun phrases as entity candidates is similar to previous work @xcite . the results are shown in though we aim to learn accurate rules, automatically acquired rules can be noisy. to ensure the quality of generated pseudo labels, we design a dynamic label selection strategy to select highly 1 noun phrases are extracted using spacy noun chunks. we evaluate our method on three datasets. experimental results show that tallor outperforms existing weakly supervised methods and can increase the average f 1 score by 60% across three datasets over methods using seed rules. further analysis shows that tallor can achieve similar performance with a state-of-the-art distantly supervised method trained using 1% of the human effort foot_0 . we also conduct a user study concerning the explainability of learned logical rules. in our study, annotators agree that 79% (on average over three annotators) of the matched logical rules can be used to explain why a span is predicted as a target entity. in summary, our main contributions are: • we define five types of logical rules and introduce compound logical rules that can accurately detect entity boundaries and classify their types. automatically learned rules can significantly reduce manual effort and provide explanations for entity predictions. we study named entity tagging under a weakly supervised setting, and propose tallor ( tagging with learnable logical rules) to build a tagger with only a small set of rules. compared with previous work, our framework requires less human unlabeled data rule candidates train & apply neural tagger score and select new rules apply rules & select training instances seed rules logical rules entity @xmath0.</@xmath1.</@xmath2.</@xmath3.</@xmath4.9 lives in (propn) → location 0.8 in (propn) → location 0.1 <@xmath5.</@xmath6.</@xmath7.</@xmath8; we also show that these rules can be used to explain tagging results. instead of treating tagging as a sequence labeling task, we formulate tagging as a span labeling task, in which named entities are modeled as spans over one or more tokens. with this setting, logical rules can easily be used for labeling entities. overview figure 2 shows the flow of our iterative learning framework, which consists of the following components. first, we generate all entity candidates and rule candidates from unlabeled data. then, for each iteration, we apply logical rules to the unlabeled data and select a set of high-quality weak training examples. next, we train a neural tagger with the selected training examples and predict the labels of unlabeled data using the trained model. finally, we select new accurate logical rules from candidate rules using the predictions. the newly learned rules will further be used to obtain weak training labels for the next iteration. ## experiments we first compare our method with baselines on three datasets and further analyze the importance of each component in an ablation study. we also report the performance of our method with different numbers of seed rules and at different iterations. finally, we show an error analysis and present a user study to analyze how many logical rules can be used as understandable explanations. ## related work different types of methods have been proposed to build named entity tagging systems using indirect or limited supervision. distant supervision @xcite is one kind of methods that have been proposed to alleviate human effort by training models using existing lexicons or knowledge bases. recently, there have been attempts to build ner systems with distant supervision @xcite @xcite . au-toner @xcite ) trained a ner system by using both typed lexicons and untyped mined phrases as supervision. @xcite proposed an adapu algorithm to incorporate an incomplete dictionary as supervision. however, lexicons or knowledge bases are not always available for new domains and tasks, especially in specific domains and low-resource settings. manually constructing these lexicons is often very expensive. bootstrapping is a technique to learn models from a small set of seeds, which has been proposed for word sense disambiguation @xcite and product attribute extraction @xcite . bootstrapping methods @xcite have been proposed for building entity tagging systems by assuming target entities are just proper names or noun phrases. @xcite used an improved pattern scoring method to bootstrap domain-specific terminologies with restricted partof-speech patterns. however, previous works only focused on disambiguating entity types by assuming target entities are given or just syntactic chunks. but, as we shown earlier, target entities often do not align well with simple syntactic chunks. bootstrapping methods that can automatically detect entity boundaries and predict their types simultaneously are desirable in real-world applications. recently, methods have been proposed to obtain weak labels by manually writing labeling functions @xcite . based on this idea, several methods @xcite have been proposed for ner by assuming the availability of a sufficient amount of handcrafted labeling functions and lexicons. however, manually designing labeling rules is challenging, which requires a significant amount of manual effort and domain expertise. our work aims to learn logical rules automatically to reduce human effort. ## conclusion in this work, we explored how to build a tagger from a small set of seed logical rules and unlabeled data. we defined five types of simple logical rules and introduced compound logical rules that are composed from simple rules to detect entity boundaries and classify their types simultaneously. we also design a dynamic label selection method to select accurate pseudo labels generated from learned rules for training a discriminative tagging model. experimental results demonstrate that our method is effective and outperforms existing weakly supervised methods.