Learning to Generate Rules for Realistic Few-Shot Relation Classification: An Encoder-Decoder Approach (2024)

Abstract: We propose a neuro-symbolic approach for realistic few-shot relation classification via rules. Instead of building neural models to predict relations, we design them to output straightforward rules that can be used to extract relations. The rules are generated using custom T5-style Encoder-Decoder Language Models. Crucially, our rules are fully interpretable and pliable (i.e., humans can easily modify them to boost performance). Through a combination of rules generated by these models along with a very effective, novel baseline, we demonstrate a few-shot relation-classification performance that is comparable to or stronger than the state of the art on the Few-Shot TACRED and NYT29 benchmarks while increasing interpretability and maintaining pliability.

## introduction in recent years, many data-driven approaches have been proposed for relation classification (rc). most of them (e.g., @xcite @xcite require extensive training data similar to that in the test set. using this data, these approaches have achieved high effectiveness in rc tasks. for example, @xcite score 91.2% f1 on re-tacred @xcite . consequently, new data-lean formulations for rc have emerged, including fewshot (fs) rc tasks such as fewrel 1.0 @xcite and fewrel 2.0 @xcite , where there are very few training examples available for each relation in the test set. however, these new datasets are unrealistic with respect to how relation classification is encountered in practice. a few of the unrealistic traits of these datasets include: a) equal distribution of instances for target relations, b) all test instances having a relation associated with them, and c) absence of common nouns and pronouns as entities. to address these issues, @xcite pro-... said marc h. morial, the former new orleans mayor who is president of the national urban league. s1: in august, baldino , who lived in west chester, pa. , ... ## related work deep learning approaches sentence-pair @xcite concatenates each support sentence with the query sentence sequentially and processes them through a bert model @xcite to predict two values: the first value measures the semantic similarity between the sentences, and the second value quantifies their dissimilarity. the method then selects the relation with support sentences most similar to the query sentence as its prediction. the similarity score for the no_relation category is obtained by taking the minimum of the set of dissimilarity scores. @xcite propose two similar techniques: nav and mnav. in nav (nota as vectors), they embed query and support sentences into the same latent space. the cosine similarity between the query sentences and support sentences is calculated through their embeddings, and the relation corresponding to the support sentence with the highest similarity is chosen as the output. the core innovation in this technique is that nota (none of the above) is also represented as an embedding and treated like a relation. mnav (or multiple nota as vectors) is similar to nav, except multiple embeddings are used to represent nota. ckpt @xcite utilizes bert to complete a prompt missing key words that indicate the predicted relation. for example, given "paris is located in france", the prompt could be "paris is the [mask] of france" for the relation "a nation's capital." additionally, it expands the vocabulary indicating each relation by leveraging external knowledge and outputs relations based on similarity. our approach outperforms these methods in three of four scenarios across the evaluated datasets and offers full interpretability and pliability. ## generating rules for relation classification as mentioned before, in relation classification, we are tasked with finding the relation between two entities (subject and object) in a sentence. a rule that identifies a relation finds a path between the subject and object entities that is peculiar to that particular relation. if the path specified by a rule is found in a given sentence, we expect that such a relation exists in that sentence. in the following sections, we discuss the various ways foot_0 we generate rules for relation classification. ## experiments and results we apply the above techniques to generate rules for support sentences in the test set, creating multiple rules per target relation for each episode. if a rule matches the query sentence, the score for the corresponding relation increases. 6 . like fs-tacred, paraphrasing yields further improvements for fs-nyt29 as well. in the 1-shot scenario, we find that paraphrasing both the query and support sentences helps us attain the second-best performance by a large margin. in the 5-shot scenario, paraphrasing only the query sentences allows us to outperform all previous work. ## qualitative analysis in this section, we present results from two types of qualitative analyses. first, we perform error anal-ysis on episodes that did not result in the correct answer when using our best method. thereafter, we investigate the pliability of the generated rules by manually editing them and evaluating the improvements in performance. ## conclusions in this work, we describe an approach for realistic few-shot relation classification (fs-rc) using rules generated with encoder-decoder language models. we also present anchor-word rules, an effective baseline to generate rules. a crucial benefit of both approaches is that they generate rules which are inherently interpretable and pliable, allowing users to easily understand and modify them. later, we evaluate the rules generated by our methods on two datasets -fs-tacred and fs- nyt29. we find that anchor-word rules are highly effective for fs-tacred outperforming most previous work, except for the state of the art. additionally, our model-generated rules excel in two of four scenarios across the datasets, surpassing most prior efforts. when combined, these rules match state-of-the-art performance in the 1-shot scenario of fs-tacred and outperform all previous methods in the 5-shot scenarios of both fs-tacred and fs-nyt29.