Summarization as Indirect Supervision for Relation Extraction (2022)

Abstract: Relation extraction (RE) models have been challenged by their reliance on training data with expensive annotations. Considering that summarization tasks aim at acquiring concise expressions of synoptical information from the longer context, these tasks naturally align with the objective of RE, i.e., extracting a kind of synoptical information that describes the relation of entity mentions. We present SuRE, which converts RE into a summarization formulation. SuRE leads to more precise and resource-efficient RE based on indirect supervision from summarization tasks. To achieve this goal, we develop sentence and relation conversion techniques that essentially bridge the formulation of summarization and RE tasks. We also incorporate constraint decoding techniques with Trie scoring to further enhance summarization-based RE with robust inference. Experiments on three RE datasets demonstrate the effectiveness of SuRE in both full-dataset and low-resource settings, showing that summarization is a promising source of indirect supervision signals to improve RE models.

## introduction relation extraction (re) aims at extracting relations between entity mentions from their textual context. for example, given a sentence "steve jobs is the founder of apple", an re model would identify the relation "founded" between mentioned entities "steve jobs" and "apple". re is a fundamental natural language understanding task and is also the essential step of structural knowledge acquisition for constructing knowledge bases. hence, advanced re models is crucial for various knowledgedriven downstream tasks, such as dialogue system @xcite , narrative prediction @xcite , and question answering @xcite . given sentences with detected pairs of entity mentions, most recent studies formulate re as multi-class classification @xcite @xcite . models presented in these studies employ pretrained language models (plm) equipped with classification heads and are finetuned with a cross-entropy loss. although such methods have achieved enhanced performances on several benchmarks @xcite @xcite , they fall short of capturing the semantic meanings of the relations. this shortage hinders plms from effectively matching the sentential context with the relations that are merely converted as logits. on the other hand, obtaining high-quality annotations for re is often costly due to the difficulty for annotators to recognize and mutually agree on such structural information. this represents another challenge for re models that have relied on direct supervision from sufficient end-task training data. existing literature finds that classification models have drastically degraded performance under low-resource scenarios @xcite , showing that label efficiency is a vital issue when adopting prior methods in real application scenarios. to combat this issue, we aim at investigating an indirectly supervised method for re, which allows the use additional supervision signals that are not specific to re without solely relying on direct re annotations. this study proposes sure (summarization as relation extraction), which reformulates and addresses re as a summarization task. 2 summarization seeks to acquire concise expressions of synoptical information from longer context @xcite , which aligns well with the objective of re if we consider the relation between entities as mandelbrot was born in poland but as a child moved to france. trie scoring output: figure 1: overview of sure inference with an example. sure constructs input sequences and verbalizes candidate relations with semantic templates (relation verbalization subfigure). then we use a summarization model to find the best prediction by trie scoring technique (trie scoring subfigure). this model is pretrained on summarization tasks and then simply finetuned with input sequence target verbalized ground-truth relation. for each common prefix (cp), we will calculate the probability of each relation candidate, which is obtained using a trained summarization model, as shown in the next token prediction subfigure. the "{subj}" and "{obj}" are two placeholders representing subject and object entity in each sample. in comparison to a multi-class classifier, summarizing the relation information also allows generating a semantically rich representation of the relation. furthermore, unlike existing re models that rely on costly manual annotations of structural information on training sentences, summarization tasks allow training with considerably richer and unannotated parallel text corpora. 3 hence, summarization tasks can bring in abundant indirect supervision signals, and can potentially lead to label-efficient models under scenarios without much task-specific annotation. fig. 1 illustrates the structure of sure. specifically, sure transforms re to a summarization task with relation and sentence conversion techniques ( §3.2), and applies constrained inference for relation prediction ( §3.4). we deploy an entity information verbalization technique to highlight the sentential contexts with entity information, and verbalize 3 particularly, summarization corpora are constructed in the scales from hundred thousands @xcite to million scales @xcite , and may be rapidly augmented in large scales from easy-to-consume data sources (e.g., community q&a platforms @xcite and scientific paper abstracts @xcite ). the contributions of this work are two-folds. first, to the best of our knowledge, this is the first study on using indirect supervision from summarization for re. since the objective of summarization naturally aligns with re, it allows precise re models to be trained without solely relying on direct task annotations, and benefits with robust re under low-resource scenarios. second, we investigate input conversion techniques that effectively bridge the formulation of summarization and re tasks, as well as constraint techniques that further enhance the inference of summarization-based re. our contributions are verified with experiments on three widely used sentence-level re datasets, ta-cred, tacrev, and semeval, as well as three low-resource settings of tacred. we observe that sure outperforms various baselines, especially in the low-resource setting with 10% tacred train-ing data. sure also achieves sota performance with 75.1% and 83.5% in micro f1 on tacred and tacrev, respectively. we also perform comprehensive ablation studies to show the effectiveness of indirect supervision from summarization and the best options of input conversion for sure. ## related work relation extraction. recent studies on (sentencelevel) re typically formulate the task as multiclass classification tasks by finetuning pretrained language models @xcite @xcite or developing pretraining objectives for re @xcite . for example, @xcite enrich the contextual representation of the plm with marked-out subject and object entity mentions. @xcite propose a model-agnostic paradigm that introduces mutual restrictions of relations and entity types into relation classifiers. on the basis of plms, some studies further improve re with external knowledge from knowledge bases (kbs) @xcite @xcite . more studies have been introduced to improve entity pair representations and classifiers for re such that we cannot exhaust them in this short summary. we refer readers to the recent benchmarking study @xcite . another threads of recent effort in re introduce several reformulations of re with prompt learning @xcite . specifically, @xcite propose prompt tuning methods for re by applying logic rules to construct hierarchical prompts. @xcite leverage prompt tuning for re by injecting semantics of relations and entity types. instead of leveraging pretrained masked language models, we use generative approaches to solve re. indirect supervision. indirect supervision @xcite methods often modify the training and inference processes on a task into a different formation, hence allowing the use of additional supervision signals that is not specific to this task. @xcite show that re can be addressed as answering reading comprehension questions and improved by the training process of a machine reading comprehensive task. similarly, @xcite also employ qa data to improve model generalization abilities in coreference resolution. @xcite propose a few-shot nli-based framework to address different tasks, such as question answering and coreference resolution. @xcite further improve this strategy by incorporating nli with learning-to-rank, leading to a robust system for ultra-fine entity typing @xcite . similar idea of leveraging nli as indirect supervision signal is applied by @xcite , which focuses on low-resource re task. as discussed, the objective of re aligns well with that of a summarization task. while there is no prior study that investigates indirect supervision from summarization, this is exactly the focus of our study. ## method in this section, we describe sure, a model for addressing re with summarization. we introduce preliminaries ( §3.1), how re data are converted to suit summarization tasks ( §3.2), training ( §3.3), and constrained inference of sure ( §3.4). ## experiments in this section, we present the experimental evaluation of sure for re under both high-and lowresource setups ( §4.1- §4.2). in addition, we also conduct comprehensive ablation studies to investigate the effectiveness of the incorporated techniques in sure ( §4.3). ## conclusion we propose a new method sure that leverages indirect supervision from summarization tasks to improve re. to do so, we verbalize relations with semantic templates, and augments entity information as parts of the linguistic context in the inputs to allow them to suit the formation of summarization. we also incorporate sure with constrained inference based on trie scoring, as well as inference with abstention and entity type constraints. extensive experiments show that such indirectly supervised re by sure lead to more precise and resource-efficient re. future work includes further developing our model on document-level re tasks and minimizing manual effort in template design with prompt tuning @xcite .