Improving Data Annotation for Low-Resource Relation Extraction with Logical Rule-Augmented Collaborative Language Models (2025)

Abstract: Low-resource relation extraction aims to identify semantic relationships between entities using scarce labeled data. Recent studies exploit large language models to recognize relations based on retrieved examplars, yielding promising results. However, the reliability of predictions from these methods is constrained by the presence of irrelevant context within demonstrations and the inherent flaws of large language models in producing undesired outputs. Inspired by the precision and generalization of abstract logic, in this paper, we propose distilling logical rules to uniformly represent task knowledge sourced from distinct origins and facilitate deductive reasoning. We develop a collaborative annotating framework that iteratively integrates high-confidence predictions of rule-enhanced relation extractors with varying scales, efficiently obtaining reliable pseudo annotations from massive unlabeled samples without human supervision. Experiments under two inference settings show that our approach achieves new state-of-the-art performance on benchmark datasets in few-shot scenarios.

## introduction the relation extraction (re) task, which aims at inferring semantic associations between recognized entities, attracts extensive research interest due to its broad range of applications, such as question answering @xcite and web mining @xcite . neural networks-based re methods rely on adequate training samples with ground-truth annotations @xcite . however, manually annotating unlabeled data sacrifices high labor costs. the scarcity of labeled data motivates the exploration of lowresource relation extraction @xcite . numerous low-resource methods are proposed to enhance the generalization capabilities of models, typically categorized into two groups: devising data-efficient models @xcite and augmenting data @xcite . with the rapid evolution of large language models (llm) @xcite , researchers increasingly employ these generative models to identify relations @xcite . the main concerns regarding using llms in production scenarios arise from their slow inference speed and huge computational demands. consequently, several works shift focus to exploiting llms as data annotators and training dedicated task models for fast processing and affordable computation overhead @xcite . in light of the dearth of ground-truth annotations impeding the re performance, in this paper, our objective is to employ the extensive knowledge embedded within the llm @xcite to harness the advantages of both humanannotated and unlabeled data. the key challenge lies in obtaining high-quality pseudo-labels from unlabeled samples. nonetheless, the reliability of predictions from prior llm-based methods is limited by two issues: (1) the mapping function of the re task is represented by demonstrations formatted as sample-label pairs, where noisy patterns within the irrelevant context can mislead llms in comprehending input-output mappings @xcite . to address the first issue, we introduce logical rules to empower the llm in maintaining concise definitions of task mapping functions. a logical labeling rule for re is defined as a premise containing critical contextual patterns and a conclusion that assigns the appropriate relation label. each rule abstracts underlying logic from the initial sample and can be adapted to other unseen samples. we use these logical rules to elicit the deductive reasoning ability of the llm, enabling more precise relation prediction. motivated by the generalization of abstract logic, we treat rules as unified explainable task knowledge learned from diverse origins, as illustrated in figure 1 . regarding the limitations of relying solely on the llm, we propose a collaborative language models-based data annotation (clma) framework that iteratively integrates rule-enhanced weak relation extractors with varied scales to acquire higher-quality pseudo labels. we initially adopt an induction-refinement procedure to assist the llm in creating consistent labeling rules from few-shot labeled instances. the induced rule set is employed to build a task-specific small language model (slm). we utilize the teacher-student framework to handle a substantial mass of unlabeled samples, where the teacher llm provides abundant knowledge and slow inference speed, whereas the task-specific student slm offers rapid inference with limited knowledge. the small task model swiftly processes unlabeled samples and exposes features for choosing candidate samples with highconfidence predictions. these selected samples are then verified by the llm through deductive reasoning guided by retrieved relevant rules. only rules from the llm that produce annotated data aligning with the high-confidence outcomes from the slm are retained to enrich the low-resource datasets. our contributions are summarized as follows: ## conclusion and future work in this paper, we propose a data annotation framework to alleviate the challenge of resource scarcity for re. logic rules are introduced to describe underlying mappings from contextual patterns to target labels. we employ the llm as a teacher to generate rule-augmented data, distilling the critical pattern grounding abilities into the small student model. high-quality pseudo-labeled samples are acquired by collaborating with distinct relation extractors. experiments suggest that our method surpasses recent competitive baselines. in future research, we are interested in extending our method to more challenging tasks like document-level re.