Discourse Understanding and Factual Consistency in Abstractive Summarization (2021)

Abstract: We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary. Our work addresses current limitations of models for abstractive summarization that often hallucinate information or generate summaries with coherence issues. To generate abstractive summaries with factual consistency and narrative flow, we propose Cooperative Generator-Discriminator Networks (Co-opNet), a novel transformer-based framework where the generator works with a discriminator architecture to compose coherent long-form summaries. We explore four different discriminator objectives which each capture a different aspect of coherence, including whether salient spans of generated abstracts are hallucinated or appear in the input context, and the likelihood of sentence adjacency in generated abstracts. We measure the ability of Co-opNet to learn these objectives with arXiv scientific papers, using the abstracts as a proxy for gold long-form scientific article summaries. Empirical results from automatic and human evaluations demonstrate that Co-opNet learns to summarize with considerably improved global coherence compared to competitive baselines.

## introduction generating summaries with coherent discourse structure and domain knowledge awareness poses a challenge for current methods in summarization. generative models can commonly produce high quality text (figure 1 ), but fail to understand finergrained details of coherence such as the structure and flow of a narrative. in addition, they often generate factually incorrect content. prior work on factuality in abstractive summarization has found that current models can hallucinate information more stochastic birth-death-immigration models are widely used in biology and ecology to study population dynamics. in this paper, we introduce a new formalism for describing... we consider the evolution of multispecies populations in which each individual is assigned a species. the model is based on birth-deathimmigration (bdiy), and we assume... we study the evolution of stochastic models for the evolution of multispecies populations, where each species is a clone of a parent species... figure 1 : generated abstracts for a biology article (from the bio subset of our arxiv dataset). abstracts are ranked from most (top) to least likely (bottom) using the generator model. abstracts with better narrative structure and domain-specific content (such as the circled abstract) are often out-ranked in terms of likelihood by abstracts with factual errors and less structure. to address these issues, we focus our study on generating abstractive summaries with factuality and narrative flow. given an input document, the goal is to generate a paragraph-length abstractive summary with proper discourse structure that contains factually correct claims. our study builds on and extends previous work that focuses on either extractive document-level summarization @xcite or abstractive sentence-level summarization @xcite @xcite . in pursuit of this goal, we introduce cooperative generator-discriminator networks (co-opnet), a framework for abstractive summarization that considers subtle aspects of fact-checking and discourse necessary for coherent text generation. in this framework, the generator, a transformer language model fine-tuned for abstractive summarization, proposes a pool of candidate summaries ( §2). the discriminator, also transformer-based, scores the factuality or discourse quality of candidate summaries using one of four different objectives: the overlap between a scientific article introduction and predicted fact-checking evidence spans in generated summaries, the ordering of predicted discourse roles, the coverage of predicted discourse roles, or the likelihood of adjacency between generated sentences ( §3). the best summary is chosen cooperatively by combining the generator and discriminator scores ( §4). most previous works on abstractive documentlevel summarization have difficulty in directly modeling or evaluating narrative flow and factuality in generated summaries. this weakness is largely due to the inherent limitations of existing datasets, such as the cnn/dailymail dataset @xcite . the reference summaries available in these commonly used resources are mainly headlines of news articles or stories. as a result, they are often sets of disconnected sentences that are highly extractive, leading to models that are also extractive @xcite , rather than abstractive. in order to address these data challenges, we test our summarization model on a set of arxiv scientific papers. scientific abstracts are ideal for modeling narrative flow as they are structured with highly coherent discourse flow. they also maintain implicit abstractive alignments with respect to the introduction of the article -in contrast to the tight, extractive alignments of current models. scientific article summarization is also a task where factuality is more well-defined than in other domains like story summarization which leave more room for interpretation. comprehensive empirical results considering both automatic and human evaluations demonstrate that co-opnet learns to summarize scientific articles from three domains with considerably improved global coherence compared to competitive baselines ( §6). we also demonstrate that the framework is generalizable to multiple coherence objectives, and effective at generating scientific abstracts that are more factually consistent. ## generator networks we use the transformer architecture of radford et al. (2019 at each time step i, the model produces an output probability distribution over the vocabulary for the next token w i given all previous output tokens @xmath0. for any arbitrary token w j preceding w i , the per-layer representation of that token is computed in the following way: ## discriminator networks because summarization models are prone to narrative flow and factual consistency issues @xcite , we use a discriminator to score generated summaries for discourse and factuality properties. due to the challenge of explicitly defining discourse and factuality properties as scores, these properties are approximated using parameterized scoring functions. these scoring functions determine if generated text demonstrates discourse and factuality properties in three ways: (1) predicting the discourse role of sentences within a full summary, (2) predicting the likelihood of adjacency given a sentence pair, and (3) measuring the presence of salient facts in the generated summary from the original input context. while our discriminators focus on these three properties, we note that this framework is generalizable and could be extended to include other generation pool of hypothesis summaries s 1 s 2 s 3 s j . . . g 1 g n . . . . . . generated tokens generated tokens summarization model probabilities p 1 , p 2 , ... , p n s 1 sentence pairs s 1 . . . s 2 s j-1 s i-1 . . . s 1 s 2 s 3 s i . . . s i s 2 s j transformer pairwise adjacency discriminator adjacency scores a 1 , a 2 , ... , a n final predicted summary reranking transformer summarization architecture input context + pos embed . . . . . . source document figure 2: model architecture for adjacency reranking variation of co-opnet discriminator models that encourage different communicative norms associated with high-quality language generation. ## reranking with discourse and factuality experts to incorporate the discriminator objective into our summarization framework, we first generate a pool of candidate summaries from the base summarization model ( §2) using any decoding strategy (e.g., beam search or top-k sampling). then, the discriminator is used to re-rank these candidates in conjunction with the original token-level generator scores. for example, in the case of the adjacency discriminator, we maximize the generator tokenlevel probability of a candidate summary g, and the average of adjacency scores for the set of sentences composing g (denoted s(g)) -i.e., the probability of each sentence s u being adjacent to the previous sentence s where λ gen and λ disc are hyper-parameters controlling the contribution of the generator and adjacency discriminator to the final predicted summary. the same procedure is followed for the other discourse and factuality objectives, replacing p adj (s u , s u-1 ) with the scores from these discriminators. ## experimental setup our implementation is based on the huggingface implementation 8 of the bert @xcite and gpt-2 language models @xcite . generator we perform wordpiece tokenization for the input context and output summaries. because of the fixed input size of the transformer language model, the input context is truncated to a maximum of 800 tokens, and summaries are truncated to a maximum of 200 tokens. we use a learning rate of 2e-5 and a batch size of 16 to finetune the generator. we the adjacency discriminator models are adapted from the huggingface implementation of the bert next sentence prediction classifier. we initialize the 12-layer bert-base discriminator model with the pretrained weights of the scibert-uncased model, which was originally trained on 1.14 million scientific papers @xcite . two discriminators are trained: one is fine-tuned on aan for decoding both arxiv cs and aan, while the other discriminator is fine-tuned on arxiv bio and used exclusively for decoding that subset. we weigh the generation and discriminator models equally when decoding by setting λ @xmath1.5. additional implementation details are provided in appendices a.3 and a.4. ## 7 experiments we compare against extractive approaches using the lede-3 and lexrank @xcite baselines. we also compare against two abstractive approaches: a 2-layer bi-lstm sequenceto-sequence model with attention (lstm), and a pointer-generator model (pgen; @xcite . training details of the supervised baselines can be found in the appendix a.2. in addition, we compare to a subset of our approach that only uses the generator to produce summaries, rather than the full framework. 10 our code/data is released here: https://github. com/skgabriel/coopnet . ## conclusion in this work, we introduced cooperative generator-discriminator networks, a framework for more coherent natural language generation with transformer language models through the integration of discriminators that encourage proper narrative flow and factual consistency. through our analyses over scientific papers from arxiv and aan, we empirically showed that our framework selects generations that are more relevant and narratively coherent than previous approaches.