Reasoning Makes Good Annotators : An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction (2023)

Abstract: Relation extraction is often challenged by insufficient labeled data. Previous methods exploit knowledge from unlabeled data by generating pseudo labels in a self-training pipeline, which suffers a gradual drift problem. Logic rules, a transferable and explainable form of expert knowledge, have achieved promising success by improving the model with weak labels. But manually writing comprehensive rules set is challenging and tedious. To alleviate the human labor of writing high-quality rules, in this work, we propose ARIA, an Automatic task-specific Rules distilling framework. Specifically, we guide the pre-trained language model to reason rules as experts and compose them into robust compound rules for data labeling. Besides, ARIA could continuously enrich the rules set to power the labeling ability by discovering reliable model-labeled data for distinguishable rules generation. Experiments on two public datasets demonstrate the effectiveness of ARIA in a low-resource scenario.

## introduction relation extraction is a fundamental task in natural language processing. training supervised models with manually annotated data is labor-intensive. this motivates methods for model learning under a low-resource setting with limited annotations. semi-supervised methods @xcite aim to explore knowledge from the unlabeled data for better model generalization. self-training pipeline @xcite iteratively adds the model's high-confidence predictions over unlabeled set to the training set and re-trains the model. however, the noise in the model-labeled data may accumulate during the training process (gradual drift problem). the logic rule is an explainable and transferred form to summarize knowledge, which could replace human for weak labels generation. since the human written rules @xcite are timeconsuming and difficult to be completed enough for emerging domains, some work attempts to generate logic rules automatically. for example, the distant supervised methods @xcite extract the knowledge base (kb)'s facts as rules for data labeling. these methods label the sentences containing the specific entity pair with kb's relation label regardless of the context, which generates noise labels easily. thus, how to handle the data's context for accurate labeling deserves studying. recently, the pre-trained language model (plm) shows broad cognitive capabilities that could be distilled into downstream tasks and work well even without any training data @xcite . specifically, some work @xcite propose the chain of thought prompts to exploit the plm's reasoning ability by guiding it to generate the intermediate natural language reasoning process like human. the explainable reasoning process infers the association between the input and output and could be deemed as the prerequisite for the output answer. motivated by this, given labeled instances as input, we guide the plm to imitate human relation reasoning manner and summarize the key information supporting relation inference from the reasoning process into transferred rules automatically. we propose an automatic task-specific logic rules distilling framework, aria, which leverages plm to replace human for continuously highquality labeling rules discovery. as shown in figure 1 , starting from limited seed data, aria alter-nates between data-to-rule stage and rule-to-data stage. the former guides the plm to reason specific rules from labeled data by following human reasoning paths and asks the plm to merge them into compound rules. the latter adopts the rules over the unlabeled set to improve the re model, which is leveraged to generalize beyond the existing rules. then we filter reliable model-labeled data for further high-quality rules generation. different from the previous work, which summarizes rules based on the restricted knowledge from the experts or knowledge bases, aria could continuously explore comprehensive rules by guiding plm to imitate human reasoning manner. there are two major challenges to automatically generate task-specific rules for accurate labeling: 1) how to guide the plm to follow human's reasoning manner and summarize crucial information of relation inference into comprehensive rules; 2) how to diversify the reasoned rules set with high-quality rules to improve the labeling ability. to solve the first issue, we guide plm to imitate typical reasoning chains of human (e.g., induction, abduction) by defining several types of meta-rule templates, which derives the key information related to the relation inference that can be used for the rules construction. for each data, to summarize rules from it, we guide the plm's reasoning by prompts, which are built by following the reasoning-specific meta-rule templates, and the output reasoning words are used to build different types of rules. since some work @xcite shows reasoning in different ways helps answer correctly, we compose the reasoning rules into compound rules and ask plm to pick out the most comprehensive compositions for robust labeling. to solve the second issue, the re model improved by the rule-labeled data, is used to generalize beyond the existing rules and alleviate the gradual drift problem. since the plm's reasoning rules conclude the information crucial for relation inference, for each relation, we pick the model-labeled data that could generate rules consistent with this relation's existing rules and distinguishable from other relations' by modeling their rules' relevance. specifically, we propose a graph-based data filter, which builds a graph of both model-labeled data and seed data to propagate their rules' features. for each relation, the model-labeled data with features close to its seed data and far from others' are picked for further rules generation. compared with previous work, our method leverages plm's broad knowledge rather than human knowledge or the restricted knowledge of jointly trained modules to discover data for high-quality rules generation. in summary, our contributions are four-fold: embedding roberta as a safe and efficient annotator, aria gets competitive labeling precision as chatgpt. we also show the smallscaled plm's potential to assist chatgpt for reliable reasoning: the in-context learning enhanced by our rules' representative information could reduce chatgpt's hallucination and improve at most 23.04% in precision. weakly supervised methods. logic rules are proposed to improve the model with weak labels. since the manually written rules are expensive and difficult to be completed enough for emerging domains @xcite , many works discover rules automatically from kb @xcite ; seed data reasoning rules generator compound rules compiler compound rules set rule-labeled data training set re model filtered data reasoning prompts ℳ reasoning rules enumerate compound rules evaluating prompts unlabeled data set add gat graph-based data filter ℳ r q q r meta-rules model-labeled data figure 2: the overall framework of aria. in each iteration, aria 1) leverages the reasoning rules generator to imitate human's reasoning manner and summarize rules from the labeled data; 2) asks the compound rules compiler to compose reasoning rules into robust compound rules; 3) utilizes compound rules to label data for training set enrichment; 4) learns a re model to predict on the unlabeled data and uses graph-based data filter to select reliable model-labeled data for further rules generation in the next iteration. ye and ling, 2019). recently, language models has shown their ability on various tasks based on prompting @xcite @xcite . since the scale of kb is limited, prboost @xcite asks plm by prompt to predict the relation directly and take the predictions as rules. this rule construction manner lacks capturing the task-specific reasoning process, making the rules less transferred and explainable. instead, aria builds rules in a fine-grained manner by guiding plm with different reasoning paths. besides, prboost requires human for rules selection, while aria could automatically pick the data that could generate high-quality rules by modeling their reasoning rules' dependency. ## method we introduce aria in this section. given limited seed data, aria continuously distills specific rules by plm for labeling over the unlabeled set and improves the task model with rule-labeled data. overview. as shown in figure 2 , our framework iterates among the four steps: 1) from data to rules: given a set of labeled data, we propose a reasoning rules generator to guide the plm to generate specific reasoning rules of different reasoning path from each labeled data. 2) from simple rules to compound rules: a compound rules compiler is proposed to compose each data's reasoning rules and ask the language model to pick the comprehensive composition as compound rules. notice that in step 4) we pick the model-labeled data rather than rule-labeled data for further rules discovery, which are less likely to accumulate repeating patterns in the rules set based on the model's generalization ability. for initialization, the seed data are taken as the input in the first iteration. then for the iteration t + 1, the input labeled data is the filtered data output from the iteration t. ## conclusion we propose aria, which guides the plm to summarize comprehensive rules as human. specifically, we build a reasoning rules generator to replace human for high-quality rules generation and a com-