Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward (2020)

Abstract: Sequence-to-sequence models for abstractive summarization have been studied extensively, yet the generated summaries commonly suffer from fabricated content, and are often found to be near-extractive. We argue that, to address these issues, the summarizer should acquire semantic interpretation over input, e.g., via structured representation, to allow the generation of more informative summaries. In this paper, we present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD. We propose the use of dual encoders—a sequential document encoder and a graph-structured encoder—to maintain the global context and local characteristics of entities, complementing each other. We further design a reward based on a multiple choice cloze test to drive the model to better capture entity interactions. Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets. We also obtain better or comparable performance compared to systems that are fine-tuned from large pretrained language models. Human judges further rate our model outputs as more informative and containing fewer unfaithful errors.

## introduction abstractive summarization aims to produce concise and informative summaries with the goal of promoting efficient information consumption and knowledge acquisition @xcite . significant progress has been made in this area by designing sequence-to-sequence-based neural models for single-document abstractive summarization @xcite @xcite . however, due to the limitations of model structure and word prediction-based figure 1 : sample knowledge graph constructed from an article snippet. the graph localizes relevant information for entities (color coded, e.g. "john m. fabrizi") or events (underlined) and provides global context. we argue that the generation of informative and succinct abstracts requires structured representation to facilitate the connection of relevant subjects, and the preservation of global context, e.g. entity interactions and topic flows. take fig. 1 as an ex-ample. complex events related with the same entity may span multiple sentences, making it challenging for existing sequential models to capture. a graph representation, on the contrary, produces a structured summary and highlights the proximity of relevant concepts. to this end, we present asgard, a framework for abstractive summarization with graph-augmentation and semantic-driven reward. 1 under the encoder-decoder framework, we enhance the regular document encoder with a separate graph-structured encoder to maintain the global context and local characteristics of entities by using the outputs from an open information extraction (openie) system. specifically, we experiment with two graph variants, one mainly capturing entities' document-level interactions and the other reflecting such interactions within each paragraph plus topic shifts across paragraphs. both graphs can capture interactions among entities that are positioned far from one another in the document and significantly reduce redundancy, as shown in fig. 1 . the document encoder and the graph encoder then cooperate during abstract generation, wherein the model is trained to identify salient content by aligning graphs with human summaries. though structured representation has been studied before for summarization @xcite , to the best of our knowledge, we are the first to utilize graph neural networks to explicitly encode entity-centered information for abstractive summary generation. moreover, we propose a novel multi-choice cloze reward to drive the model to acquire semantic understanding over the input. concretely, we design cloze questions by removing pairwise entities that are connected with a predicate or co-occur in a human summary sentence, whereas prior work only considers single entities to construct questions @xcite . in tandem with our graph encoding of knowledge, the cloze reward further facilitates the acquisition of global entity interactions with reinforcement learning. we carry out automatic and human evaluations on popular summarization datasets. models based on asgard yield significantly better rouge scores @xcite than a variant without access to the knowledge graph on two popular news summarization datasets, new york times the rest of the paper is organized as follows. we describe related work in the next section ( § 2). we then discuss the knowledge graph construction in § 3 and formulate our graph-augmented summarization framework in § 4. in § 5, we introduce reinforcement learning with cloze reward. experiments and results are presented in § 6 and § 7. finally, we conclude in § 8. ## related work graph-augmented summarization and generation. graph structures have long been used for extractive summarization, such as in textrank @xcite and lexrank @xcite . for neural models, tan et al. (2017) design graph-based attention to identify important sentences. for generating abstractive summaries, fernandes et al. (2019) enhance a sequence-based encoder with graph neural networks (gnns) to consider token-level entity types, however, entity interactions are largely ignored. on multi-document summarization, fan et al. (2019) demonstrate the usefulness of encoding a linearized knowledge graph from openie outputs. in this work, we design a graph encoder, which improves upon graph attention networks (gats) @xcite , to capture the global context in a more effective manner. also related is the graph-to-sequence framework that has been adopted for text generation @xcite . both gated graph neural networks (ggnns) @xcite and graph convolutional networks (gcns) @xcite are shown to be effective in generating sentences from amr graphs. since graph attention networks can better handle sparse graphs, they are used by @xcite with a transformer model to create scientific paper ab-stracts from knowledge graphs. here we use graphs in addition to document encoder, both carrying complementary information for summarization. reinforcement learning and qa reward for abstractive summarization. as pointed out by @xcite , word-level maximum likelihood training brings the problem of exposure bias. recent work utilizes reinforcement learning to directly optimize the model to maximize the informativeness of summaries by using different forms of rouge scores @xcite @xcite . however, rouge does not always distinguish good summaries from bad ones @xcite , and ignores entity interactions. since question answering (qa) has been used for summary evaluation @xcite , and is shown to correlate with human judgment of summaries qualities @xcite , qa-based rewards have been studied for summarization model training. @xcite demonstrate that using fill-in-the-blank questions by removing entities or root words leads to improved content selection. @xcite consider a similar setup, but use both f1 score and qa system confidence as rewards in abstractive summarization. previous work, however, mainly focuses on single entities or words in human-written summaries, thereby losing contexts and relations. moreover, fill-in-the-blank questions by prior work give credits only when the answers exactly match the ground-truths, thus causing inaccuracies for rephrased answers and discouraging abstract content generation. in contrast, we design a semantic-driven cloze reward by measuring how well a qa system can address multiple choice cloze questions which better encode entity interactions and handle paraphrased answers. ## knowledge graph construction to construct a knowledge graph from an input document, we utilize stanford corenlp @xcite to first obtain outputs from coreference resolution and open information extraction (openie) models @xcite . note that we do not conduct global entity linking across documents. next, we take the subject, predicate, object triples extracted by openie and remove any triple whose argument (subject or object) has more than 10 words. if two triples differ only by one argument, and the arguments overlap, we keep the longer triple. ## summarization model in this section, we describe our graph-augmented abstractive summarization framework, as displayed in fig. 2 . our model takes as input a document, represented as a sequence of tokens @xmath0, and a knowledge graph g consisting of nodes {v i }. x and g are separately consumed by a document encoder and a graph encoder, as presented in § 4.1. importantly, we present two types of graphs: doc-graph, focusing on the global context, and seg-graph, which additionally captures topic shift. the summary decoder then generates an abstractive summary by attending to both the document and the graph ( § 4.2). in § 4.3, we formulate a maximum likelihood training objective which leverages the detection of salient nodes in the graph. ## reinforcement learning with cloze after maximum likelihood training with l ml , we further design a multiple choice cloze reward in a second-stage reinforcement learning (rl), leading the model to generate more faithful and informative summaries. for rl, we use a self-critical policy gradient algorithm @xcite . during training, two summaries are generated: first, a summary y s , sampling tokens based on the probability distribution p(y s | x; θ) at each decoding step; and second, a baseline summary ŷ which greedily selects the tokens of the highest probability at each step. the objective of rl is defined based on the rewards of the two summaries, r(y s ) and r(ŷ), as follows: our reward function uses the combination of rouge and the multiple choice cloze score introduced below, i.e., r(y) = r rouge (y) + γ cloze r cloze . for rouge, it considers f1 scores of rouge-1, rouge-2, and rouge-l calculated against the reference summary, and takes the form of r rouge (y multiple choice cloze reward. here, we present a novel multiple choice cloze reward to work with our knowledge graph and guide the summarization model towards improved awareness of entity interactions. we treat the systemgenerated summary as context. we provide a set of questions automatically constructed from the corresponding reference summary written by a human. we separately train a question answering (qa) model to address the questions by reading the context. intuitively, if the system summary shares salient information with the reference, the qa model will assign the correct answers with high probability. we decide to use the average probability of the correct answers as our cloze reward. below, we give details on how to construct the questions and candidate answers with examples shown in fig. 3 . question construction. we run the openie tool on human-written summaries, retaining triples with arguments not longer than 5 words. for each triple of subject, predicate, object , we create two types of questions: (1) argument pair questions, by removing the subject and object, and (2) predicate questions, by removing the predicate. candidate answer construction. because fill-inthe-blank style cloze may incorrectly penalize qa systems with answers paraphrased from the groundtruth, we opt for a multiple choice cloze. we construct three candidate answers in addition to the reference summary: federal reserve increases interest rates. ie output: federal reserve, increases, interest rates salient context: federal reserve signals positivity about the market. fed increases benchmark interest rate again this may. american economy keeps the high growth rate. jerome h. powell discussed potential risks. ie outputs: 1. federal reserve, signals, positivity 2. american economy, keeps, the high growth rate 3. jerome h. powell, discussed, potential risks ⇓ multiple choice cloze questions: argument pair question: increases . a. federal reserve, interest rates ( ) b. interest rates, federal reserve (swapping args in a) c. american economy, interest rates (replacing arg using triple 2) d. federal reserve, potential risks (replacing arg using triple 3) predicate question: federal reserve interest rates. a. increases ( ) b. signals c. keeps d. discussed gold-standard from the salient context, which are summary-worthy sentences selected from the input. specifically, we use greedy search to select the best combination of sentences that maximizes rouge-2 f1 with reference to human summary. we further include a sentence in the salient context if it has a rouge-l recall greater than 0.6 when compared with any sentence in the reference. we first select openie triples from the salient context and filter out those that have any overlapping content word with the correct answer. for argument pair questions, we create one candidate answer by swapping the subject and the object (e.g. candidate b as in fig. 3 ) and two candidates by replacing the subject or the object with another argument of the same role extracted from the salient context (e.g. candidates c and d). if not enough answers are created, we further consider randomly selecting sentences from the input. for predicate questions, we use predicates in other triples from the context as candidate answers. among all candidates, we select the three that are able to construct the most fluent questions using perplexity predicted by bert @xcite . in case reference summaries do not yield ope-nie triples, we create additional entity pair questions. we remove two co-occurring entities from the summary and create three candidate answers in the same way as described above. qa model. we fine-tune roberta @xcite to build our qa model. we use the salient context described above as the context for training. we then concatenate the context, the question, and each of the four candidate answers, and pass the final [cls] representation through a fully-connected layer, from which the answer is predicted. ## experimental setups datasets. we experiment with two popular summarization datasets with summaries containing multiple sentences: the new york times annotated corpus (nyt) @xcite and the cnn/daily mail dataset (cnn/dm) @xcite . we follow the preprocessing steps and experimental setups from prior work @xcite for both datasets. for nyt, the training, validation, and test sets contain 588, 909, 32, 716, and 32, 703 samples. for cnn/dm, the numbers are 287, 188, 13, 367, and 11, 490. to train our cloze qa model for nyt, we construct 1, 414, 336 question-answer pairs from human-written summaries in the training set based on the method described in § 5. on cnn/dm, we collect 1, 361, 175 question-answer samples from the training set. for both datasets, we set aside 20, 000 samples as a validation set and 20, 000 samples as a test set. our qa model achieves an accuracy of 97% on nyt and 95% on cnn. training details and parameters. we use the base version of roberta model to extract token features for all experiments. we truncate input articles to 1024 (nyt) and 512 (cnn/dm) bpes. we employ lstm models with 256-dimensional hidden states for the document encoder (128 each direction) and the decoder. for the residual connection of the graph encoder, we use 4 heads, each with a dimension of 72. for docgraph training and inference, we prune isolated graphs with fewer than three nodes to increase robustness and reduce redundancy. we set γ 1 = 0, γ 2 = 0.75 on nyt and γ 1 = 0.33, γ 2 = 0.33 on cnn/dm after tuning on the validation set. for both datasets, we set γ @xmath1.05. more details about parameters and graph statistics are in the appendices. baselines and comparisons. for both datasets, we include an extractive baseline lead-3. we further add the following abstractive models for comparison: (1) a pointer-generator model with coverage @xcite (pointgen+cov); (2) a deep reinforcement learning-based model (paulus et al., 2018) (deepreinforce); (3) a bottom-up model (gehrmann et al., 2018) (bottomup); ( 4 ) a deep communicating agents-based summarization model @xcite . we also report results by fine-tuning bart model @xcite . in @xcite , fine-tuning is only performed on cnn/daily mail. we apply the same method for nyt. for nyt, we add results by seneca model @xcite ) from our prior work, which previously achieved the best rouge-2. on cnn/daily mail, we include comparisons of a two-stage fine-tuned model (first on an extractor, then on an abstractor) with bert (liu and lapata, 2019) (bertsumextabs), and a unified pretrained language model for generation @xcite (unilm). in addition to asgard-doc and asgard-seg, which are trained with an ml objective, we report results trained with rouge as the reward (r rouge ), and with an additional cloze reward (r cloze ). lastly, we consider a variant nograph by ablating the graph encoder. ## conclusion we presented a novel knowledge graph-augmented abstractive summarization framework, along with a novel multiple choice cloze reward for reinforcement learning. our models capture both local characteristics and global interactions of entities from the input, thus generating summaries of higher quality. in tandem with the graph representation, our cloze reward further improves summary content. human evaluation further confirms that our graphaugmented models trained with the cloze reward produce more informative summaries and significantly reduces unfaithful errors.