Automatic Rule Induction for Efficient Semi-Supervised Learning (2022)

Abstract: Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Meanwhile, pretrained transformer models act as black-box correlation engines that are difficult to explain and sometimes behave unreliably. In this paper, we propose tackling both of these challenges via Automatic Rule Induction (ARI), a simple and general-purpose framework for the automatic discovery and integration of symbolic rules into pretrained transformer models. First, we extract weak symbolic rules from low-capacity machine learning models trained on small amounts of labeled data. Next, we use an attention mechanism to integrate these rules into high-capacity pretrained transformer models. Last, the rule-augmented system becomes part of a self-training framework to boost supervision signal on unlabeled data. These steps can be layered beneath a variety of existing weak supervision and semi-supervised NLP algorithms in order to improve performance and interpretability. Experiments across nine sequence classification and relation extraction tasks suggest that ARI can improve state-of-the-art methods with no manual effort and minimal computational overhead.

## introduction large-scale pretrained neural networks can struggle to generalize from small amounts of labeled data @xcite , motivating approaches that leverage both labeled and unlabeled data. this is partially due to the black-box and correlational nature of neural networks, which confers the additional difficulties of uninterpretability @xcite and unreliability @xcite . a growing body of research seeks to ameliorate these issues by augmenting neural networks with symbolic components: heuristics, logical formulas, program traces, network templating, blacklists, etc @xcite @xcite . in this paper, we refer to these components as rules. symbolic reasoning has attractive properties. rules need little or no data to systematically generalize, and rules are inherently interpretable with respect to their constituent operations. in this paper we propose a general-purpose framework for the automatic discovery and integration of symbolic rules into pretrained models. the framework contrasts with prior neuro-symbolic nlp research in two ways. first, we present a fully automatic rule generation procedure, whereas prior work has largely focused on manually crafted rules @xcite @xcite or semi-manual rule generation procedures @xcite @xcite . with these existing techniques, practioners must formulate and implement their rules by hand, creating a second-order "rule annotation" burden on top of the data labeling process. second, the proposed framework is general purpose and can be applied to any classification dataset. this contrasts with prior research that proposes task-and domain-specific symbolic logic, through weak supervision signals @xcite @xcite , special loss functions @xcite , model architectures @xcite , and prompt templates @xcite . our framework consists of two steps. first, we generate symbolic rules from data. this involves training low-capacity machine learning models on a reduced feature space, extracting artifacts from these models which are predictive of the class labels, then converting these artifacts into rules. after the rule induction step, we use the induced rules to amplify training signal in the unlabeled data. in particular, we adopt a rule-augmented self-training procedure, using an attention mechanism to aggregate the predictions of a backbone classifier (e.g. bert) and the rules. we evaluate the ari framework across nine text classification and relation extraction tasks. the results suggest that the proposed algorithm can exceed state-of-the-art semi-supervised baselines, and that these gains may be because the model learns to rely more heavily on rules for difficult-topredict examples. we also show that the proposed rule induction strategy can rival human crafted rules in terms of their quality. last, we demonstrate the interpretabiltiy of the overall system. in summary, the contributions of this paper are: 1 ## the ari framework the proposed rule induction framework seeks to automatically induce symbolic rules from labeled data. next, the rules can be used to amplify training signal on the unlabeled data. these steps are depicted in fig. 1 . more formally, assume we are given a target classification task consisting of labeled classification data @xmath0, y i )} m @xmath1, where each x i is a text string and y @xmath2, ..., k}. our proposed method uses the labeled data l to generate a set of symbolic prediction functions ("rules") @xmath3: r j (x) ∈ {-1} ∪ {1, ..., k}. we then train a joint system which models p (y|x; l, u, r), i.e., an estimator which utilizes the labeled data, unlabeled data, and rules to make reliable and interpretable predictions. ## experiments we perform experiments across 9 datasets and tasks, finding that the ari rule induction framework can improve the performance of state-of-theart semi-supervised text classification algorithms. note that concrete examples of the human-readable rules succeeding (and failing) are given in the appendix. ## interpretability as discussed in section 2.2, the behavior of the aggregation layer a(•) can be traced to individual rules, which are themselves human readable and interpretable. this is because the output of a(•) is a linear combination of attention scores and rule predictions (equation 1 ). in other words, if the attention score for rule r j on example x i is s j i , then the strength of rule r j 's contribution to the model's final prediction is exactly s j i /q. see the appendix for case studies showing the impact of individual rules on model behavior. to further demonstrate the system's interpretability, we grouped examples according to their difficulty 4 and measured the cumulative effect of rules 4 following @xcite , we used the en-on model behavior (i.e., j s j i /q) for each category. the results are given in ## related work our research draws on a number of related areas of research, including neuro-symbolic computation, semi-supervised learning, and weak supervision. neuro-symbolic approaches seek to unite symbolic ai, which from the 1950's until the mid 1990's was the dominant paradigm of ai research @xcite , with statistical machine learning and neural networks. for example, there is work that uses discrete parses to template neural network components @xcite @xcite . there is also work that seeks to embed symbolic knowledge into network parameters via special loss functions @xcite or carefully curated datasets @xcite @xcite and architectures @xcite . other related work seeks to incorporate logical constraints into text generation models @xcite . our framework is further inspired by semisupervised learning research that leverages labeled and unsupervised data. our baseline pet model comes from a family of algorithms that leverage prompting and model ensembling for greater data efficiency @xcite . there is also research on pulling in demonstration examples from the training set @xcite , automatic prompt generation @xcite tropy of berts predicted label distribution as a measure of example difficulty. we ranked examples according to this measure, then split them into hard (above the 75th percentile), medium (25-75th percentile) and easy (below 25th percentile). liang, 2021), and leveraging extra datasets and tasks for data augmentation when data is scarce @xcite . our self-training approach is similar to the knowledge distillation literature @xcite where a "student" model is trained to imitate the predictions of a "teacher" model. in our case, the teacher is not a separate model but a frozen student plus rule aggregation layer. another close body of research taps into weak sources of supervision like regular expressions, keywords, and knowledge base alignment @xcite @xcite . researchers have incorporated these weak supervision signals into self-training procedures like ours @xcite , as well as constructing procedural generators for boosting weak supervision signals @xcite and interactive pipelines for machine-assisted rule construction @xcite @xcite . there is also research on automatically generating weak labeling functions @xcite which shares our bag-of-words featurization and regression scoring mechanism. ## conclusion in this paper, we proposed automatic rule induction (ari), a simple and general-purpose framework for the automatic discovery and integration of symbolic rules into pretrained nlp models. our results span nine sequence classification and relation extraction tasks and suggest that ari can improve state-of-the-art algorithms with no manual effort and minimal computational overhead. future work could investigate layering ari beneath other few-shot and semi-supervised algorithms, and improving the underlying rule generation strategies, particularly with causal mechanisms @xcite .