Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
Abstract
Sequence-to-sequence (seq2seq) neural models have been actively investigated for abstractive summarization.
Nevertheless, existing neural abstractive systems frequently generate factually incorrect summaries and are vulnerable to adversarial information, suggesting a crucial lack of semantic understanding.
In this paper, we propose a novel semantic-aware neural abstractive summarization model that learns to generate high quality summaries through semantic interpretation over salient content.
A novel evaluation scheme with adversarial samples is introduced to measure how well a model identiﬁes off-topic information, where our model yields signiﬁcantly better performance than the popular pointer-generator summarizer.
Human evaluation also conﬁrms that our system summaries are uniformly more informative and faithful as well as less redundant than the seq2seq model.
Introduction
Automatic text summarization holds the promise of alleviating the information overload problem (Jones et al., 1999).
Considerable progress has been made over decades, but existing summarization systems are still largely extractive—important sentences or phrases are identiﬁed from the original text for inclusion in the output (Nenkova et al., 2011).
Extractive summaries thus unavoidably suffer from redundancy and incoherence, leading to the need for abstractive summarization methods.
Built on the success of sequence-to-sequence (seq2seq) learning models (Sutskever et al., 2014), there has been a growing interest in utilizing a neural framework for abstractive summarization (Rush et al., 2015; Nallapati et al., 2016; Wang and Ling, 2016; Tan et al., 2017; Chen and Bansal, 2018).
Although current state-of-the-art neural models naturally excel at generating grammatically correct sentences the model structure and learning objectives have intrinsic difﬁculty in acquiring semantic interpretation of the input text, which is crucial for summarization.
Importantly, the lack of semantic understanding causes existing systems to produce unfaithful generations .
Cao et al.
(2018) report that about 30% of the summaries generated from a seq2seq model contain fabricated or nonsensical information.
Furthermore, current neural summarization systems can be easily fooled by off-topic information.
For instance, Figure 1 shows one example where irrelevant sentences are added into an article about “David Collenette’s resignation”.
Both the seq2seq attentional model (Nallapati et al., 2016) and the popular pointer-generator model (See et al., 2017) are particularly susceptible to unfaithful generation, partially because these models tend to rely on sentences at the beginning of the articles for summarization while being ignorant about their content.
Therefore, we design a novel adversarial evaluation metric to measure the robustness of each summarizer against small amounts of randomly inserted topic-irrelevant information.
The intuition is that if a summarization system truly understands the salient entities and events, it would ignore unrelated content.
Article Snippet : For years Joe DiMaggio was always introduced at Yankee Stadium as “baseball’s greatest living player.”
But with his memory joining those of Babe Ruth, Lou Gehrig, Mickey Mantle and Miller
Huggins.
Canada’s Minister of Defense resigned today, a day after an army ofﬁcial testiﬁed that top military ofﬁcials had altered documents to cover up responsibility for the beating death of a Somali teen-ager at the hands of Canadian peacekeeping troops in 1993.
Defense minister David Collenette insisted that his resignation had nothing to do with the Somalia scandal.
Ted Williams was the ﬁrst name to come to mind, and he’s the greatest living hitter. ...
Seq2seq : George Vecsey sports of The Times column on New York State’s naming of late baseball legend Joe DiMaggio as “baseball’s greatest living player,” but with his memory joining those of Babe Ruth, Lou
Gehrig, Mickey Mantle and Miller dens.
Pointer-generator : Joe DiMaggio is ﬁrst name to come to mind, and Ted Williams is ﬁrst name to come to mind, and he’s greatest living hitter; he will be replaced by human resources minister, Doug Young, and will keep his Parliament seat for governing Liberal Party.
Our Model : Former Canadian Defense Min David Collenette resigns day after army ofﬁcial testiﬁes that top military ofﬁcials altered documents to cover up responsibility for death of Somali
-ager at of troops in
To address the above issues, we propose a novel semantic-aware abstractive summarization model , inspired by the human process of writing summaries—important events and entities are ﬁrst identiﬁed, and then used for summary construction.
Concretely, taking an article as input, our model ﬁrst generates a set of summary-worthy semantic structures consisting of predicates and corresponding arguments (as in semantic parsing), then constructs a ﬂuent summary reﬂecting the semantic information.
Both tasks are learned under an encoder-decoder architecture with new learning objectives.
A dual attention mechanism for summary decoding is designed to consider information from both the input article and the generated predicate-argument structures.
We further present a novel decoder with a segment-based reranking strategy to produce diverse hypotheses and reduce redundancy under the guidance of generated semantic information.
Evaluation against adversarial samples shows that while performance by the seq2seq attentional model and the pointer-generator model is impacted severely by even a small addition of topic-irrelevant information to the input, our model is signiﬁcantly more robust and consistently produces more on-topic summaries (i.e. higher ROUGE and METEOR scores for standard automatic evaluation).
Our model also achieves signiﬁcantly better ROUGE and METEOR scores than both models on the benchmark dataset CNN/Daily Mail (Hermann et al., 2015).
Speciﬁcally, our model’s summaries use substantially fewer and shorter extractive fragments than the comparisons and have less redundancy, alleviating another common problem for the seq2seq framework.
Human evaluation demonstrates that our model generates more informative and faithful summaries than the seq2seq model.
Related Work
To discourage the generation of fabricated content in neural abstractive models, a pointer-generator summarizer (See et al., 2017) is proposed to directly reuse words from the input article via a copying mechanism (Gu et al., 2016).
However, as reported in their work (See et al., 2017) and conﬁrmed by our experiments, this model produces nearly extractive summaries.
While maintaining their model’s rephrasing ability, here we improve the faithfulness and informativeness of neural summarization models by enforcing the generation of salient semantic structures via multi-task learning and a reranking-based decoder.
Our work is in part inspired by prior abstractive summarization work, where the summary generation process consists of a distinct content selection step (i.e., what to say) and a surface realization step (i.e., how to say it) (Wang and Cardie, 2013; Pighin et al., 2014).
Our model learns to generate salient semantic roles and a summary in a single end-to-end trained neural network.
Our proposed model also leverages the recent successes of multi-task learning (MTL) as applied to neural networks for a wide array of natural language processing tasks (Luong et al., 2015; Søgaard and
Goldberg, 2016; Peng et al., 2017; Rei, 2017; Isonuma et al., 2017).
Most recent work (Pasunuru et al., 2017) leverages MTL to jointly improve performance on summarization and entailment generation.
Instead of treating the tasks equally, we employ semantic parsing for the sake of facilitating more informative and faithful summary generation.
[Figure]
Model 3.1 Model Formulation
In the standard seq2seq model, a sequence of input tokens x = { x 1 , ..., x n } is encoded by a recurrent neural network (RNN) with hidden states h
i at every timestep i .
The ﬁnal hidden state of the encoder is treated as the initial state of the decoder, another RNN ( h n = s 0 ).
The decoder will produce the output sequence y =
{ y 1 , ..., y T } , with hidden states s t at every timestep t .
Our model outputs two sequences: the sequence of semantic role tokens y s =
{ y s 1 , ..., y sT s } is generated ﬁrst, followed by the sequence of summary (or abstract)
tokens y
a = { y a 1 , ..., y aT a } .
The attention mechanism of Bahdanau et al.
(2014) is utilized to attend the input.
Concretely, a context vector c inp t is calculated as the summation of the encoder hidden states, weighted by the attention distribution a t at every timestep t :
[Equation]
[Equation] [Equation]
(3) where v , W s , W h are learnable parameters.
The context vector, along with the decoder hidden state, is used to produce the vocabulary distribution via softmax:
[Equation]
[Equation]
where W v and d are learned parameters and (cid:107) represents vector concatenation.
The loss is deﬁned as the negative log likelihood of generating summary y over the training set D using model parameters θ :
[Equation]
The log probability for each training sample is the average log likelihood across decoder timesteps.
Encoder.
In our models, our encoder is a single-layer bidirectional long short-term memory (LSTM) unit (Graves and Schmidhuber, 2005), where the hidden state is a concatenation of the forwards and backwards LSTMs:
[Equation]
Decoders.
We propose two different decoder architectures— separate decoder and shared decoder — to handle semantic information.
In the separate decoder model (See Figure 2c), the semantic decoder and the summary decoder are each implemented as their own single-layer LSTM.
This setup, inspired by the one-to-many multi-task learning framework of Luong et al. (2015), encourages each decoder to focus more on its respective task.
Decoder output and attention over the input are calculated using
Eqs.
1-5, but separate sets of parameters are learned for each decoder: v s , W ss , W sh , W sv , d s for the semantic decoder, and v a , W as , W ah , W av , d a for the summary decoder.
We further study a shared decoder model (See Figure 2a) for the purpose of reducing the number of parameters as well as increasing the summary decoder’s exposure to semantic information.
One single-layer LSTM is employed to sequentially produce the important semantic structures, followed by the summary.
Our output thus becomes y =
[ y s (cid:107) y a ] , and the ﬁrst timestep of the summary decoder is the last timestep of the semantic decoder.
Attention is calculated as in Eqs.
For both models, the loss becomes the weighted sum of the semantic loss and the summary loss:
[Equation]
In our experiments, we set α as 0 .
5 unless otherwise speciﬁed.
We also investigate multi-head attention (Vaswani et al., 2017) over the input to acquire different language features.
To our knowledge, we are the ﬁrst to apply it to the task of summarization.
As shown later in the results section, this method is indeed useful for summarization, with different heads learning different features.
In fact, the multi-head attention is particularly well-suited for our shared decoder model, as some heads learn to attend semantic aspects and others learn to attend summary aspects.
Target Semantic Output.
We use semantic role labeling (SRL) as the target semantic representation, which identiﬁes predicate-argument structures.
To create the target data, articles in the training set are parsed in the style of PropBank (Kingsbury and Palmer, 2002) using the DeepSRL parser from He et al.
We then choose up to ﬁve SRL structures that have the most overlap with the reference summary.
Here we ﬁrst consider matching headwords of predicates and arguments.
If no match is found, we consider all content words.
Note that the semantic labels are only used for model training.
At test time, no external resource beyond the article itself is used for summary decoding.
3.2 Dual Attention over Input and Semantics
To further leverage semantic information, a dual attention mechanism is used to attend over the generated semantic output in addition to the input article during summary decoding.
Although attending over multiple sources of information has been studied for visual QA (Nam et al., 2017) and sentence-level summarization (Cao et al., 2018), these are computed over different encoder states .
In contrast, our dual attention mechanism considers decoding results from the semantic output.
Although our attention mechanism may appear close to intra- or self-attention (Sankaran et al., 2016), the function of our dual attention is to attend over a speciﬁc portion of the previously generated content that represents its own body of information, whileas traditional self-attention is predominantly used to discourage redundancy.
The context vector attending over the semantic decoder hidden states is calculated as follows:
[Equation]
[Equation]
where s a are the summary decoder’s hidden states, s s are the semantic decoder’s hidden states, and t (cid:48) are the time steps for the summary decoder.
The summary output o a t (cid:48) now becomes:
[Equation]
3.3 Reranking-based Summary Decoder
Albeit a standard practice, the beam search algorithm has been shown to produce suboptimal results in neural text generation problems (Vijayakumar et al., 2016) due to the fact that one beam often dominates others, yielding hypotheses that only differ by the last token.
To combat this issue, others have proposed beam rerankers that utilize external features such as Maximum Mutual Information (Li et al., 2016), segment-reranking (Shao et al., 2017), and hamming loss (Wen et al., 2015).
Here, we discuss our own segment-reranking beam search decoder, which encourages both beam diversity and semantic coverage .
Our reranker is applied only during summary decoding, where we rely on the generated semantic roles for global guidance.
The only modiﬁcation made to the semantic decoder is a de-duplication strategy, where generated predicates seen in previous output are eliminated.
Reranking.
Regular beam search chooses the hypotheses for the next timestep solely based on conditional likelihood (i.e., p ( y a | x ) ).
In our proposed summary decoder, we also leverage our generated semantic information and curb redundancy during beam selection.
Reranking is performed every R timesteps based on the following scorer, where hypotheses with less repetition ( r ) and covering more content words from the generated semantics ( s ) are ranked higher:
[Equation]
[Equation]
[Equation]
where LRS is the longest repeating substring in the current hypotheses.
We deﬁne a repeating substring as a sequence of three or more tokens that appears in that order more than once in the hypothesis, with the intuition that long repeating fragments (ex.
“ the Senate proposed a tax bill; the Senate proposed a tax bill. ”) should be penalized more heavily than short ones (ex.
“ the Senate proposed a tax bill; Senate proposed. ”).
s measures the percentage of generated semantic words reused by the current summary hypothesis, contingent on the predicate of semantic structure matching.
At every other timestep, we rank the hypotheses based on conditional likelihood and a weaker redundancy handler, r (cid:48) , that considers unigram novelty (i.e., percentage of unique content words): score = log ( p ( y a | x )) + α (cid:48) r (cid:48) .
We use R = 10 , α = 0 .
4 , β = 0 .
1 , α (cid:48)
= 0 .in
our experiments.
Beam Diversity.
In the standard beam search algorithm, all possible extensions to the beams are considered, resulting in a comparison of B × D hypotheses, where B is the number of beams and
D is the vocabulary size.
In order to make the best use of the reranking algorithm, we develop two methods to enforce hypothesis diversity during non-reranking timesteps.
Beam Expansion.
Inspired by Shao et al.
(2017), we rank only the K highest scoring extensions from each beam, where K < B .
This ensures that at least two unique hypotheses from the previous timestep will carry on to the next timestep.
Beam Selection.
We further reinforce hypothesis diversity through the two-step method of (1) likelihood selection and (2) dissimilarity selection.
In likelihood selection, we accept N hypotheses (where N < B ) from our pool of B × K based solely on conditional probabilities, as in traditional beam search.
From the remaining hypotheses pool, we select B − N hypotheses on the basis of dissimilarity.
We choose the hypotheses with the highest dissimilarity score ∆([ h ] N , h (cid:48) ) , where h (cid:48) is a candidate and [ h ] N are the hypotheses chosen during likelihood selection.
In our experiments, we use token-level Levenshtein edit distance (Levenshtein, 1966) as the dissimilarity metric, where ∆([ h ] N , h (cid:48) )
= max h ∈
[ h ] N (Lev( h, h (cid:48) )) .
In experiments, we use B = 12 , K = 6 , N = 6 .
Experimental Setup
Datasets.
We experiment with two popular large datasets of news articles paired with human-written summaries: the CNN/Daily Mail corpus (Hermann et al., 2015)
(henceforth CNN/DM) and the New
York Times corpus (Sandhaus, 2008) (henceforth NYT).
For CNN/DM, we follow the experimental setup from See et al.
and obtain a dataset consisting of 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs.
For NYT, we removed samples with articles of less than 100 words or summaries of less than 20 words.
We further remove samples with summaries containing information outside the article, e.g., “[AUTHOR]’s movie review on...” where the author’s name does not appear in the article.
NYT consists of 280,146 training pairs and 15,564 pairs each for validation and test.
Training Details and Parameters.
For all experiments, a vocabulary of 50k words shared by input and output is used.
Model parameters and learning rate are adopted from prior work (See et al., 2017) for comparison purpose.
All models are also trained in stages of increasing maximum token lengths to expedite training.
The models trained on the NYT dataset use an additional ﬁnal training stage, where we optimized only on the summary loss (i.e., α = 1 in Eq. 7).
During decoding, unknown tokens are replaced with the highest scoring word in the corresponding attention distribution.
Baselines and Comparisons.
We include as our extractive baselines T EXTRANK (Mihalcea and
Tarau, 2004) and LEAD -2, the ﬁrst 2 sentences of the input article, simulating the average length of the target summaries.
We consider as abstractive comparison models (1) vanilla seq2seq with attention ( SEQ 2 SEQ ) and (2) pointer-generator (See et al., 2017) (P OINT -G EN ), which is trained from scratch using the released code.
Results for variants of our model are also reported.
Automatic Evaluation Metrics.
For automatic evaluation, we ﬁrst report the F1 scores of ROUGE-1, 2, and L (Lin and Hovy, 2003), and METEOR scores based on exact matching and full matching that considers paraphrases, synonyms, and stemming (Denkowski and Lavie, 2014).
We further measure two important aspects of summary quality: extractiveness —how much the summary reuses article content verbatim, and redundancy —how much the summary repeats itself.
For the ﬁrst aspect, we utilize the density metric proposed by Grusky et al.
(2018) that calculates “the average length of extractive fragments”:
[Equation]
where A represents the article, S represents the summary, and F ( A, S ) is a set of greedily matched extractive fragments from the article-summary pair.
Based on density, we propose a new redundancy metric:
[Equation]
f
[Equation]
where F (cid:48)
( S ) contains a set of fragments at least three tokens long that are repeated within the summary, and # f (cid:48) is the repetition frequency for fragment f (cid:48) .
Intuitively, longer fragments and more frequent repetition should be penalized more heavily.
Results
Adversarial Evaluation.
Our pilot study suggests that the presence of minor irrelevant details in a summary often hurts a reader’s understanding severely, but such an error cannot be captured or penalized by the recall-based ROUGE and METEOR metrics.
In order to test a model’s ability to discern irrelevant information, we design a novel adversarial evaluation scheme, where we purposely mix a limited number of off-topic sentences into a test article.
The intuition is that if a summarization system truly understands the salient entities and events, it would ignore unrelated sentences.
We randomly select 5,000 articles from the CNN/DM test set with the “news” tag in the URL (mainly covering international or domestic events).
For each article, we randomly insert one to four sentences from articles in the test set with the “sports” tag.
For NYT, we randomly select 5,000 articles from the test set with government related tags (“U.S.”, “Washington”, “world”) and insert one to four sentences from articles outside the domain (“arts”, “sports”, “technology”).
We ensure that sentences containing a pronoun in the ﬁrst ﬁve words are not interrupted from the previous sentence so that discourse chains are unlikely to be broken.
the content of the article, it would ignore unrelated sentences.
We took 15,000 articles from the NYT corpus with government related tags (“u.s.”, “washington”, “world”), and for each article, randomly inserted up to four sentences from articles outside the domain (“arts”, “sports”, “technology”).
We ensured sentences that contained a pronoun in the ﬁrst ﬁve words were not interrupted from the previous sentence so as not to break discourse or coreference chains
[Figure] [Figure]
METEOR Dens.
Red.
ROUGE
METEOR Dens.
Red.
RR-
R-
[Table]
exact
[Table]
RR-
R-
[Table]
exact
[Table]
Abstractive Comparisons
[Table]
The adversarial evaluation results on seq2seq, pointer-generator and our shared decoder model are shown in Figure where our model consistently yields signiﬁcantly better ROUGE-L and METEOR scores than the comparisons, and is less affected as more irrelevant sentences are added.
Sample summaries for an adversarial sample are shown in Figure 1.
We ﬁnd that our semantic decoder plays an important role in capturing salient predicates and arguments, leading to higher quality summaries.
Automatic Evaluation.
The main results are displayed in Table 1.
On CNN/DM, all of our models signiﬁcantly outperform seq2seq across all metrics and our shared decoder model with multi-head attention yields signiﬁcantly better ROUGE (R-1 and R-L) scores than the pointer-generator on the same dataset (approximate randomization test, p < 0 . 01 ).
Despite the fact that both ROUGE and
METEOR favor recall and thus reward longer summaries, our summaries that are often shorter than comparisons still produce signiﬁcantly better ROUGE and METEOR scores than seq2seq on the
NYT dataset.
Furthermore, our system summaries, by all model variations, reuse fewer and shorter phrases from the input (i.e., lower density scores) than the pointer-generator model, signifying a potentially stronger ability to rephrase.
Note that the density scores for seq2seq are likely deﬂated due to its inability to handle out-of-vocabulary words.
Our models also produce less redundant summaries than their abstractive comparisons.
Human Evaluation.
We further conducted a pilot human evaluation on 60 samples selected from the NYT test set.
For each article, the summaries by the seq2seq model, our shared decoder model, and the human reference were displayed in a randomized order.
Two human judges, who are native or ﬂuent English speakers, were asked to read the article and rank the summaries against each other based on non-redundancy, ﬂuency, faithfulness to input, and informativeness (whether the summary delivers the main points of the article).
Ties between system outputs were allowed but discouraged.
As illustrated in Table 2, our model was ranked signiﬁcantly higher than seq2seq across all metrics.
Surprisingly, our model’s output was ranked higher than the reference summary in 26% of the samples for informativeness and ﬂuency.
We believe this is due to the speciﬁc style of the NYT reference summaries and the shorter lengths of our summaries: the informal style of the reference summaries (e.g., frequently dropping subjects and articles) negatively affects its ﬂuency rating [Table] and readers ﬁnd our shorter summaries to be more concise and to the point.
[Table]
Discussion
Usage of Semantic Roles in Summaries.
We examine the utility of the generated semantic roles.
Across all models, approximately 44% of the generated predicates are part of the reference summary, indicating the adequacy of our semantic decoder.
Furthermore, across all models, approximately 65% of the generated predicates are reused by the generated summary, and approximately 53% of the SRL structures are reused by the system using a strict matching constraint, in which the predicate and head words for all arguments must match in the summary.
When gold-standard semantic roles are used for dual attention in place of our system generations, ROUGE scores increase by about half a point, indicating that improving semantic decoder in future work will further enhance the summaries.
Coverage.
We also conduct experiments using a coverage mechanism similar to the one used in See et al.
We apply our coverage in two places: (1) over the input to handle redundancy, and (2) over the generated semantics to promote its reuse in the summary.
However, no signiﬁcant difference is observed.
Our proposed reranker handles both issues in a more explicit way, and does not require the additional training time used to learn coverage parameters.
Alternative Semantic Representation.
Our summarization model can be trained with other types of semantic information.
For example, in addition to using the salient semantic roles from the input article, we also explore using SRL parses of the reference abstracts as training signals, but the higher level of abstraction required for semantic generation hurts performance by two ROUGE points for almost all models, indicating the type of semantic structure matters greatly for the ultimate summarization task.
For future work, other semantic representation along with novel model architecture will be explored.
For instance, other forms of semantic representation can be considered, such as frame semantics (Baker et al., 1998) or Abstract Meaning Representation (AMR) (Banarescu et al., 2013).
Although previous work by Vinyals et al.
(2015) has shown that seq2seq models are able to successfully generate linearized tree structures, we may also consider generating semantic roles with a hierarchical semantic decoder (Sordoni et al., 2015).
Conclusion
We presented a novel semantic-aware neural abstractive summarization model that jointly learns summarization and semantic parsing.
A novel dual attention mechanism was designed to better capture the semantic information for summarization.
A reranking-based decoder was proposed to promote the content coverage.
Our proposed adversarial evaluation demonstrated that our model was more adept at handling irrelevant information compared to popular neural summarization models.
Experiments on two large-scale news corpora showed that our model yielded signiﬁcantly more informative, less redundant, and less extractive summaries.
Human evaluation further conﬁrmed that our summaries were more informative and faithful than comparisons.