Interactive Machine Teaching by Labeling Rules and Instances (2024)

Abstract: Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert’s limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.

## introduction supervised machine learning models for text classification require large, hand-labeled training datasets, which are both expensive and time-consuming to obtain. most efforts to reduce the reliance on large training datasets support just a single type of expert supervision, namely, to label individual instances one at a time @xcite @xcite * work done at columbia prior to joining amazon. @xcite @xcite @xcite . to reduce the data labeling bottleneck, weakly supervised learning (wsl) @xcite focuses on labeling rules that automatically generate weak labels for unlabeled instances. wsl works in two separate steps: (i) experts provide labeling rules; and (ii) labeling rules are used to train a machine learning model. most work focuses on solving the second step and learn with noisy rules @xcite @xcite @xcite . in practice, however, experts find it difficult to define sufficiently many rules in one shot @xcite . considerable time and creativity are required for inspecting unlabeled instances and creating rules that add predictive value by effectively covering a substantial number of instances. therefore, it is an open question whether experts should spend their limited time writing rules or instead providing instance labels, notably via active learning @xcite . in this paper, we investigate how to efficiently exploit an expert's limited time for machine teaching. our main idea is to automatically extract labeling rules with high coverage of unlabeled data, and then rely on domain expertise to validate the candidate rules. in contrast to active learning methods, where the machine queries the expert for labels of individual examples @xcite , providing feedback for each rule leads to multiple data labels, which we show here can boost classification performance faster. supporting rich forms of interaction is challenging, especially when the teaching budget is limited. first, given a restricted number of rules that can be created or validated by an expert, it is not clear what properties these rules should have to train an accurate model. for example, should one prioritize rules that cover many examples but with relatively low precision, or rules that have high precision but lower coverage? moreover, existing algorithms for rule extraction require substantial labeled data, and it is unclear how to extract and rank candidate rules when we are given just limited labeled data and perhaps a few expert-validated rules. in general, there are few guidelines in the literature for creating effective rules for efficient machine teaching. additionally, the option to ask for feedback on both rules and instances requires balancing the costs and potential benefits of each type of feedback when there is a shared budget of expert interaction. our work addresses these open questions via the following contributions: ## problem definition and related work we now define our problem of focus (section 2.1); we also discuss related work on non-interactive weak supervision and interactive learning with instance-and feature-level feedback (section 2.2). ## interactive machine teaching with instance and rule feedback this section describes our interactive machine teaching framework, which addresses the problem defined in section 2.1. the core question is how to efficiently solicit expert feedback for machine teaching given a limited budget t . our main idea is to balance the quality of instance labels with the efficiency of labeling rules under this low-resource setting. we propose a framework, interval, that supports efficient interaction by selecting which instances to label manually and by extracting candidate rules that, when accepted, can automatically generate many additional labels. interval can be used with several wsl methods and any learning model. in the rest of this section, we describe the individual steps followed by interval on each iteration, namely, teacher-student co-training (section 3.1), querying for instance feedback (section 3.2), candidate rule extraction (section 3.3), and querying for rule feedback (section 3.4), and then we summarize the main ideas of our interactive machine teaching algorithm (section 3.5). ## experimental settings we now present our experimental setting for interactive machine teaching on several text classification datasets. ## experimental results we now present our analysis of expert-provided rules (section 5.1), results on automatic rule extraction (section 5.2), and our experiments for interactive machine teaching with queries on instances and rules (section 5.3). ## discussion and future work our framework and analysis demonstrates the advantages of soliciting feedback on both candidate rules and individual instances. we identify several areas for future research and discuss them next. as future work, we will explore additional design choices for interval, including instance selection strategies (e.g., based on rule informativeness), rule extraction methods (e.g., based on rule diversity), and weak supervision techniques. while interval selects up to β candidate rules per instance (where β i depends on how many rules satisfy the precision and coverage thresholds), we could further explore adaptive querying protocols, for example dynamically determining β or selectively skipping instance labeling based on dataset characteristics or labeling costs. we could also extend interval to support richer types of feedback, such as editing (rather than accepting or rejecting) candidate rules and prompt templates (rather than relying on fixed templates from @xcite ). more research is required from a user perspective, for example on how to visualize rules @xcite and effectively present a combination of rules and instances for expert labeling. inter-val supports prompting pre-trained models just for training data creation, and can work with any model for inference, thus enabling applications where deploying large language models might not be possible. we expect further gains by creating rules using more powerful pre-trained models such as instructgpt @xcite ; palm-t5 @xcite ; llama @xcite . we also expect performance improvements by replacing the student using stronger pre-trained models and by representing instances using more recent text embedding techniques @xcite @xcite . interval could also be extended for multi-label classification by changing the teacher-student co-training objective (section 3.1) and for other broader tasks by generating rules from more complex rule families using models such as toolformer @xcite . our current experimental evaluation used simulated expert feedback, because a definitive evaluation involving actual subject matter experts would be too expensive. a potential stopgap is to use large language models (such as chatgpt), which may be too expensive to query at test time, but are cheaper than subject matter experts to query at training time for selected instances. ## conclusions in this paper, we presented an interactive machine teaching approach that queries experts for feedback on both instances and automatically generated rules. our findings show that, even though rules are domain specific and have diverse characteristics, there are patterns that are prevalent across datasets. specifically, a higher-f1 teacher does not necessarily lead to a higher-f1 student. we identified that the teacher's precision is more important than coverage for training an accurate student. these findings could potentially inform guidelines for rule creation. our analysis demonstrates that automatic rules based on high-level predicates are more accurate than rules based on n-gram predicates. we additionally showed that by asking queries on both instances and automatically extracted rules, our method can be more effective than active learning methods.