Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification (2024)

Abstract: This paper introduces a novel neuro-symbolic architecture for relation classification (RC) that combines rule-based methods with contemporary deep learning techniques. This approach capitalizes on the strengths of both paradigms: the adaptability of rule-based systems and the generalization power of neural networks. Our architecture consists of two components: a declarative rule-based model for transparent classification and a neural component to enhance rule generalizability through semantic text matching.Notably, our semantic matcher is trained in an unsupervised domain-agnostic way, solely with synthetic data.Further, these components are loosely coupled, allowing for rule modifications without retraining the semantic matcher.In our evaluation, we focused on two few-shot relation classification datasets: Few-Shot TACRED and a Few-Shot version of NYT29. We show that our proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data.Further, we show that our approach remains modular and pliable, i.e., the corresponding rules can be locally modified to improve the overall model. Human interventions to the rules for the TACRED relation org:parents boost the performance on that relation by as much as 26% relative improvement, without negatively impacting the other relations, and without retraining the semantic matching component.

## introduction after the "deep learning tsunami" @xcite , neural approaches for information extraction (ie) consistently pushed the boundaries of the state of the art @xcite @xcite @xcite . however, all these directions come at a cost: rule [@xmath0: an example of the type of rules we use in our proposed method, together with three sentences. the rule captures the org:founder relation with a syntactic pattern anchored by the predicate founded that has a person named entity as its subject and an organization as the direct object. by itself, the rule matches the first sentence, but it does not match the other two. when coupled with our semantic matching component, the rule matches the first two sentences. explainability is critical in many domains such as healthcare, law, and finance @xcite @xcite . while there have been efforts to incorporate explainability into neural methods @xcite @xcite , most explanations are local and post-hoc, which has two important drawbacks. first, such explanations are not guaranteed to be faithful @xcite . second, they are not actionable. that is, it is not immediately possible to modify the underlying model using insights from the explanations without risking introducing new, unforeseen behavior. in contrast, rule-based foot_1 methods are explainable and pliable, 3 but lack the generalization power of current deep learning systems @xcite . in this paper, we propose a novel neuro-symbolic architecture for relation classification (rc) that preserves the advantages of both directions, i.e., the generalization of neural methods and the pliability of rule-based approaches with a modular ap-proach, containing two components: a declarative rule-based model and a neural component. the first module implements relation classification with a set of explainable rules. the second increases the generalizability of rules by semantically matching them over text. figure 1 shows an example of how the two components interact. our specific contributions are: ## related work we overview the three main directions that influenced this work -rule-based approaches, bootstrapping or other seed-based approaches, and explainable deep learning methods-as well as differences between the proposed work and prompting/incontext learning. ## proposed method we propose a hybrid model that combines the advantages of rule-based and neural approaches. our approach first attempts to strictly match rules, i.e., all semantic/syntactic/lexical constraints must match in the input sentence for a match to be considered. if no rule matches, we back off to a neural semantic rule matching (srm) component that semantically aligns rules with text. a key aspect of our proposed approach is that we do not incorporate a no_relation classifier in any form, such as a nav or mnav @xcite . this is important as training multiple representation vectors to capture the entire no_relation space, as proposed in @xcite can be difficult in practice, as reported by the original authors. instead, our method is simpler: we have rules with associated underlying relations and a single threshold @xmath1, 1] to decide whether the srm assigned score between a rule and a sentence constitutes a match or not. this threshold is application-specific and can be selected on a development set. ## , 11 we concentrate our discussion on comparing between contemporary rule-based methods (odin-synth) and strong neural-based methods (mnav). we draw the following observations. first, compared to mnav, the state-of-the-art neural-based method on few-shot tacred, our proposed approach outperforms it in three out of the four settings investigated. for example, in the 1-shot case of few-shot tacred, our proposed method improves upon mnav by over 12 f1 points (approximately 100% relative improvement), despite not being trained with any human-annotated data or with any tacred-specific data. we remark that mnav outperforms our proposed approach in the 1-shot case on few-shot nyt29. nyt29 was annotated using distant supervision, which often results in shallow, context-free patterns. our analysis indicates that mnav, due to its training approach, may be effectively capturing these simple entity patterns. for example, for a sentence such as "barack obama was born in honolulu .", we hypothesize that mnav might superficially link (barack obama, honolulu) to the relation "born in", irrespective of the context. consequently, mnav could mistakenly assign the same relation to a contextually different sentence like "barack obama went to high school in honolulu", where the entities remain the same but the relation differs. we manually checked the top ten most popular entities from the support sentences and from the test sentences and observed that all have corresponding wikipedia pages (i.e., they are very frequent), further supporting our hypothesis. second, our hybrid method largely surpasses odinsynth, the leading rule-based approach on few-shot tacred, in both 1-shot and 5-shot scenarios. this validates the hypothesis that combining a neural network with traditional rule-based approaches outperforms rule-only methods. the improved performance of our method does not sacrifice precision; it significantly surpasses odinsynth in both precision and recall. this conclusion also applies to the few-shot variant of nyt29. all in all, our proposed method obtains state-of-the-art performance despite not being trained on any of the human-annotated examples from the respective training datasets. ## conclusion we introduced a novel neuro-symbolic approach for relation extraction that combines the better generalization of neural networks with the explainability and pliability of rules. our method first attempts to match the rule in a typical binary match/no match way. when a rule does not match, our approach then semantically matches it over text using a semantic matching component, which is contrastively trained without any human-annotated training data, akin to an llm for rules. we evaluated our model on two challenging fewshot datasets: few-shot tacred @xcite and a few-shot variant of nyt29 @xcite . we showed that our method achieves strong performance, outperforming state-of-the-art supervised methods in three out of the four settings we investigated. moreover, we empirically validated that our proposed method retains the pliability of rule-based methods, i.e., where humans can refine the underlying classification rules to noticeably increase the final performance. notably, the resulting model is relatively small, i.e., it consists of an encoder of approximately 350m parameters, which makes it considerably more efficient than a decoder-based llm.