Improving Zero-shot Relation Classification via Automatically-acquired Entailment Templates (2023)

Abstract: While fully supervised relation classification (RC) models perform well on large-scale datasets, their performance drops drastically in low-resource settings. As generating annotated examples are expensive, recent zero-shot methods have been proposed that reformulate RC into other NLP tasks for which supervision exists such as textual entailment. However, these methods rely on templates that are manually created which is costly and requires domain expertise. In this paper, we present a novel strategy for template generation for relation classification, which is based on adapting Harris’ distributional similarity principle to templates encoded using contextualized representations. Further, we perform empirical evaluation of different strategies for combining the automatically acquired templates with manual templates. The experimental results on TACRED show that our approach not only performs better than the zero-shot RC methods that only use manual templates, but also that it achieves state-of-the-art performance for zero-shot TACRED at 64.3 F1 score.

## introduction relation classification (rc) identifies the relation that holds between two entities that co-occur in the same text. for example, given the sentence: jane's white house desk has a sign "home on the range", the relation between jane and white house is employee_of. beyond this simple example, rc is a critical nlp task with important applications to many domains such as intelligence @xcite and biomedical @xcite . recent directions mitigate the amount of supervision necessary for rc by taking advantage of the knowledge stored in large language models (llms). for example, @xcite reformulated rc as an entailment task based on templates that are manually created as the verbalizations of relation labels. then they are used to formulate a hypothesis that can be verified with an off-theshelf llm entailment engine. other directions feed prompts that capture the definition of the task and examples into encoder-decoder language models @xcite . however, most of these directions tend to rely on templates/prompts that are manually-created by domain experts. this strategy has a potential high cost, and also runs the risk of inserting undesired biases in the data. directions that focus on automatically learning prompts often produce prompts that are nonsensical to humans @xcite our work aims to limit the above drawbacks of template-based approaches for rc. in particular, we expand the approach of @xcite with explainable templates that are automatically acquired from a large textual collection. for template acquisition, we modify the bert-informed rule discovery (bird) algorithm @xcite , which, given a seed template, automatically generates templates with similar meaning. bird encodes templates using contextualized representations generated by a transformer network @xcite , and then uses a similarity measure to acquire similar templates based on an extension of the distributional similarity principle (harris, 1954) to templates. for example, given a manual template {subj} was founded by {obj}, the template {subj} was created by {obj} is automatically created. we use these automatically-acquired templates to expand the pool of manual templates used for rc, and show that this expansion yields statistically significant performance improvements. the key contributions of this paper are: our work is different in that we focus on the zeroshot scenario, a common situation in the real world, where a rc system must be developed rapidly without the luxury of training data. ## background as mentioned, our zero-shot approach for relation classification relies on an entailment engine that is fed automatically-generated templates. in this section we discuss the building blocks necessary for this idea. ## approach one limitation of @xcite ) is that they rely on manually-generated templates, which require effort to create and may prone to bias. our work mitigates this limitation by automatically expanding the manual patterns using bird. we utilize the manual templates as seed patterns to feed them into bird to generate new patterns that tend to have similar meanings to the manual templates. figure 2 shows this process. for each manual template, we extract its syntactic path. for example, for the template {subj} was founded by {obj}, we extract the syntactic path nsubjpass←@xmath0. then the syntactic path is fed to bird to generate new syntactic paths. we generate 40 patterns for each manual template. after the syntactic paths are generated, we generate their corresponding english text using a module we created. for example, for the syntactic path nsubjpass←@xmath1, the english text "x was created by y" is generated. the corresponding patterns of 12 manual templates did not exist in bird's pattern collection. therefore, it was not possible to use them as seed patterns. for these templates we manually selected an existing high-quality pattern that was very close in meaning. for example, the pattern "x is spouse of y" did not exist in bird's pattern collection. we chose the pattern "x married y" instead. ## conclusions this paper introduces a zero-shot approach for relation classification. our direction expands the method of @xcite , which converts the relation extraction task into textual entailment that is informed by manual templates that characterize the relations of interest. unlike @xcite , we combine manual templates with templates that were automatically-acquired using an adaption of harris' distributional similarity principle to templates encoded using contextualized representations. we empirically evaluate our approach on a zeroshot setting of the tacred relation classification task @xcite . we investigated multiple strategies to rank the quality of the automaticallyacquired templates. all in all, we found that a simple strategy, which considers the top template as ranked by a textual entailment engine, performs the best. our results match the current state-ofthe-art for zero-shot tacred. we find this result exciting, especially considering that the template acquisition component is disconnected from the tacred dataset. beyond these results, this paper opens interesting questions for future work such as how to increase the relevance of the automaticallyacquired templates for a specific task.