R elation P rompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction (2022)

Abstract: Despite the importance of relation extraction in building and representing knowledge, less research is focused on generalizing to unseen relations types. We introduce the task setting of Zero-Shot Relation Triplet Extraction (ZeroRTE) to encourage further research in low-resource relation extraction methods. Given an input sentence, each extracted triplet consists of the head entity, relation label, and tail entity where the relation label is not seen at the training stage. To solve ZeroRTE, we propose to synthesize relation examples by prompting language models to generate structured texts. Concretely, we unify language model prompts and structured text approaches to design a structured prompt template for generating synthetic relation samples when conditioning on relation label prompts (RelationPrompt). To overcome the limitation for extracting multiple relation triplets in a sentence, we design a novel Triplet Search Decoding method. Experiments on FewRel and Wiki-ZSL datasets show the efficacy of RelationPrompt for the ZeroRTE task and zero-shot relation classification. Our code and data are available at github.com/declare-lab/RelationPrompt.

## introduction relation extraction aims to predict relationships between entities in unstructured text, which has applications such as knowledge graph construction @xcite and question answering @xcite . however, existing approaches often require large datasets of annotated samples which are costly to annotate and have a fixed set of relations. currently, less research is focused on the zero-shot setting @xcite where models need to generalize to unseen relation sets without available annotated samples @xcite . although there are existing zero-shot relation task settings, they do not require extracting the full relation triplets. the task setting of zero-shot relation classification foot_0 (zerorc) was previously introduced by @xcite to classify the relation between a given head and tail entity pair for unseen labels. however, it is not always practical or realistic to assume that the ground-truth entities are readily available. zero-shot relation slot-filling @xcite aims to predict the tail entity based on the provided head entity and relation, but also relies on other methods for entity detection. thus, it also faces the challenge of error propagation in practice @xcite . hence, we propose a new and challenging task setting called @xcite . the goal of zerorte is to extract triplets of the form (head entity, tail entity, relation label) from each sentence despite not having any annotated training samples that contain the test relation labels. for a clear comparison between task settings, we provide a summary in to solve the challenges of data scarcity, there are several existing approaches. although distant supervision @xcite can be used to construct a relation corpus with a many relation types, this approach generally results in lower annotation quality than human annotation. furthermore, distant supervision remains limited to a fixed set of relation types in the existing knowledge base @xcite . another approach is to formulate the task objective such that the label space is unconstrained. for instance, zero-shot sentence classification can be reframed as entailment @xcite or embedding similarity (pushp and srivastava, 2017) objectives. however, the existing formulations are designed for sequence classification tasks, which cannot be directly applied to structured prediction tasks such as relation triplet extraction. a third direction is to leverage pre-trained language models using task-specific prompt templates @xcite which enables the models to generalize to new tasks with little to no training samples, such as zero-text classification @xcite . this zero-shot potential is possible by leveraging the semantic information in prompts to query the language comprehension capabilities of pre-trained language models @xcite . hence, we propose relationprompt which reframes the zero-shot problem as synthetic data generation. the core concept is to leverage the semantics of relation labels, prompting language models to generate synthetic training samples which can express the desired relations. the synthetic data can then be used to train another model to perform the zero-shot task. this capability is supported by the finding that language models can be prompted to control task-specific aspects of the generated text, such as domain and content @xcite . for instance, given the relation label "military rank" in figure 1c , it is reasonable to condition the language model and compose a sentence demonstrating the relationship that a person has been bestowed with a certain position in the armed forces. hence, a possible sentence could be "she is the wife of lieutenant colonel george hocham.", ## related work zero-shot relation extraction zero-shot relation extraction was previously framed as a slotfilling task and solved by reading comprehension methods @xcite . however, their approach requires manual template design for each relation label, which cannot scale well to new relation types. another approach to zero-shot relation extraction is the formulation into an entailment task @xcite , which is not constrained to a fixed relation label space. instead, the entailment approach determines if arbitrary pairs of sentences and relation labels are compatible. however, it is designed for sentence classification and cannot be applied to zerorte. 125 250 500 1000 2000 22 24 26 28 30 generated samples per label f 1 (%) figure 6: effect of generated data size on zerorte. data augmentation a popular method for improving model performance in supervised lowresource tasks is data augmentation. simple heuristics such as token manipulation @xcite were initially developed, new methods in language modeling improved the quality of augmented samples @xcite . although there are data augmentation methods that can be applied to structured tasks such as named entity recognition @xcite and relation extraction @xcite , they require existing training samples and cannot be easily adapted to zero-shot tasks. knowledge retrieval relationprompt also leverages the knowledge stored in language models @xcite to compose relation samples that are grounded in realistic contexts. to ensure that the generated samples are factually accurate, the language model requires strong knowledge retrieval capabilities @xcite . language model prompts prompting-based methods have shown promise as a new paradigm for zero-shot or few-shot inference in natural language processing @xcite . another advantage is the potential to adapt very large language models @xcite to new tasks without relatively expensive fine-tuning. con-current works @xcite also show that language models can generate synthetic training data. however, such methods have not yet proven effective for more complex tasks such as triplet extraction. structured prediction relationprompt generates synthetic data for relation triplet extraction, which is a structured prediction task. hence, it can be widely applicable to other structured prediction tasks such as named entity recognition @xcite , event extraction @xcite or aspect sentiment triplet extraction @xcite . ## conclusions and future work triplet extraction (zerorte) to overcome fundamental limitations in previous task settings and encourage further research in lowresource relation extraction. to solve zerorte, we propose relationprompt and show that language models can effectively generate synthetic training data through relation label prompts to output structured texts. to overcome the limitation for extracting multiple relation triplets in a sentence, we propose the triplet search decoding method which is effective and interpretable. results show that our method surpasses prior zerorc methods as well as strong baselines on zerorte, setting the bar for future work. as mentioned in section 4.3, a future direction for improvement could be to ensure that the generated entity spans are more compatible with the semantics of the relation in the language model prompt.