S UMMARIZATION P ROGRAMS : I NTERPRETABLE A BSTRACTIVE S UMMARIZATION WITH N EURAL M ODULAR T REES
A BSTRACT
Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales by only highlighting parts of the source document.
To this end, we propose the Summarization Program (SP) , an interpretable modular framework consisting of an (ordered) list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence from the source document.
A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence (root node) to the document sentences (leaf nodes) from which it is derived, with the connecting nodes containing intermediate generated sentences.
Edges represent different modular operations involved in summarization such as sentence fusion, compression, and paraphrasing.
We first propose an efficient best-first search method over neural modules, SP-S EARCH that identifies SPs for human summaries by directly optimizing for ROUGE scores.
Next, using these programs as automatic supervision, we propose seq2seq models that generate Summarization Programs, which are then executed to obtain final summaries.
We demonstrate that SP-S EARCH effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior.
We also conduct a simulation study to show that Summarization Programs improve the interpretability of summarization models by allowing humans to better simulate model reasoning .
Summarization Programs constitute a promising step toward interpretable and modular abstractive summarization, a complex task previously addressed primarily through blackbox end-to-end neural systems.
I NTRODUCTION
Progress in pre-trained language models has led to state-of-the-art abstractive summarization models capable of generating highly fluent and concise summaries (Lewis et al., 2020;
Zhang et al., 2020; Raffel et al., 2020).
Abstractive summarization models do not suffer from the restrictive nature of extractive summarization systems that only copy parts of the source document.
However, their ability to generate non-factual content (Cao et al., 2018; Maynez et al., 2020) and their lack of clear interpretability makes it harder to debug their errors and deploy them in real-world scenarios.
Towards interpretable summarization models, Jing & McKeown (1999; 2000) show that human summaries typically follow a cut-and-paste process, and propose a modular architecture involving separate operations that perform sentence extraction, sentence reduction, sentence fusion, etc.
Most recent efforts on explainable abstractive summarization follow an extractive-abstractive framework that only provides supporting evidence or ‘rationales’ for the summary (Hsu et al., 2018; Gehrmann et al., 2018; Liu & Lapata, 2019;
Zhao et al., 2020; Li et al., 2021).
These models highlight words or sentences from the source document but are not able to explicitly capture the generative process of a summary, i.e., the reasoning steps performed in order to generate each summary sentence from the source document sentence(s), like sentence compression, fusion, etc. In this work, we seek to bridge [Figure] Summarization Program
Edges directed from leaves to roots, and the intermediate generations are labeled with I1 and I2.
this gap by proposing a novel Summarization Program framework for explaining abstractive summarization, that views summarization as a systematic reasoning process over document sentences.
A Summarization Program (SP) is a modular executable program that consists of an (ordered) list of binary trees, each encoding the generative process of an abstractive summary sentence from the source document (§3).
Fig.
1 shows an example. The leaves in an SP are the source document sentences (typically, only a small subset that are relevant for generating the summary). Each intermediate node represents a generation from a neural module (shown with labeled edges) which are then composed to derive the final summary sentences at the roots of the trees.
We develop three neural modules for building SPs – sentence compression, paraphrasing, and fusion (Jing & McKeown, 1999; 2000) that finetune a pre-trained language model on task-specific data.
We evaluate Summarization Programs by asking the following two research questions (see Fig.
2 for an overview).
1. RQ1.
Given a human-written abstractive summary, can we develop an algorithm for identifying a Summarization Program that effectively represents the generative process of the summary?
RQ2.
Using the SPs identified in RQ1 as supervision, can we develop models that generate Summarization Programs as interpretable intermediate representations for generating summaries?
We answer the first research question by automatically identifying SPs for human summaries (§4).
Specifically, we develop an efficient best-first search algorithm, SP-S EARCH that iteratively applies different neural modules to a set of extracted document sentences in order to generate newer sentences, such that the ROUGE (Lin, 2004) score with respect to the gold summary is maximized by these new sentences.
SP-S EARCH achieves efficiency through important design choices including maintaining a priority queue that scores, ranks, and prunes intermediate generations (appendix A).
We conduct experiments on two English single document summarization datasets, CNN/DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018) to show that SP-S EARCH outputs SPs that effectively reproduce human summaries, significantly outperforming several baselines (§6.1).
Moreover, human evaluation shows that our neural modules are highly faithful, 2 performing operations they are supposed to and also generating outputs that are mostly factual to their inputs (§6.2).
We leverage SP-S EARCH to obtain oracle programs for human summaries that also serve as supervision for answering our second research question.
In particular, we propose two seq2seq models for Summarization Program generation from a source document (§5, Fig. 2).
In our first Extract-andBuild SP generation model, an extractive summarization model first selects a set of document sentences, which are then passed to another program-generating model.
In our second Joint SP generation model, sentence extraction and SP generation happen as part of a single model.
We obtain initial promising results and while state-of-the-art end2end models demonstrate better ROUGE scores, our oracle SP-S EARCH results indicate significant room for improvement in future work (§6.3).
To evaluate whether SPs improve the interpretability of summarization models, we conduct a smallscale simulation study (Doshi-Velez & Kim, 2017; Hase & Bansal, 2020;
Zhou et al., 2022) where we ask humans to simulate model’s reasoning by writing programs for model summaries (§6.4).
We observe that after seeing model SPs, humans are able to better predict them for unseen samples, such that the executed summaries match more closely with the model summaries.
Our contributions are:
• We introduce the Summarization Program , an interpretable modular framework for explainable abstractive summarization.
Summarization Program consists of an ordered list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence through the use of different neural modules (sentence compression, paraphrasing, and fusion).
• We propose an efficient best-first search method, SP-S EARCH that identifies Summarization Programs for human-written summaries, obtaining a high ROUGE-2 of 40 on the CNN/DailyMail dataset with neural modules that are highly faithful to their intended behavior.
• We present initial Summarization Program generation models that generate SPs from a source document, which are then executed to obtain final summaries.
We demonstrate that SPs improve the interpretability of summarization models by allowing humans to better simulate model behavior.
R ELATED W ORK
Pre-trained Models for Summarization
State-of-the-art pre-trained language models like
BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and PEGASUS (Zhang et al., 2020) generate summaries in an end-to-end manner.
Lacking complete mechanistic understanding of transformers (Elhage et al., 2021), it remains difficult to understand the reasoning process behind the generated summary.
In particular, we cannot determine whether generation follows a process similar to that of humans, e.g., using operations like selection of important information, abstraction through vocabulary generalization, sentence fusion, etc (Kintsch & van Dijk, 1978; Brown & Day, 1983;
Jing &
McKeown, 1999; 2000).
Moreover, it is hard to determine the source of errors in summaries without knowing the reasoning path to the summary.
Recent work on interpretable summarization models highlight words or sentences as rationales for the generated summaries (Hsu et al., 2018; Gehrmann et al., 2018; Liu & Lapata, 2019;
Liu et al., 2019; Zhao et al., 2020;
Li et al., 2021)
, but textual highlights do not provide a complete explanation of the summary generation process.
Thus, Summarization Program is a step towards more explicit interpretability in abstractive summarization.
Neural Module Networks.
Our work also draws inspiration from Neural Module Networks (NMN) that execute programs as learned functions composed of neural modules (Andreas et al., 2016;
Hu et al., 2018; Jiang & Bansal, 2019; Gupta et al., 2020; Subramanian et al., 2020; Saha et al., 2021a;
Le et al., 2022).
Typically, the modules in an NMN provide attention-based explanations whose interpretability has been debated (Serrano & Smith, 2019; Wiegreffe & Pinter, 2019).
Khot et al.
(2021) alternatively propose Text Modular Networks that decompose multi-hop questions into subquestions to be solved by simpler QA models.
We also follow this text-in text-out paradigm for our modules but equipped to perform diverse sentence-level operations in document summarization.
We evaluate our modules for ‘neural module faithfulness’ (Subramanian et al., 2020) to demonstrate that the SPs mostly provide a faithful interpretation of the generated summaries.
Multi-step Reasoning over Text.
Step-by-step reasoning has received much interest as a way to explain various reasoning tasks including QA (Dalvi et al., 2021;
Ribeiro et al., 2022) and natural language deduction (Bostrom et al., 2022; Saha et al., 2020; Gontier et al., 2020; Saha et al., 2021b; Tafjord et al., 2021).
Summarization Programs similarly encode the reasoning steps in a summary generation process.
Our SP-S EARCH method follows a forward chaining method that tries to reach a hypothesis from a set of premises (Tafjord et al., 2021; Bostrom et al., 2022), unlike backward chaining methods that do the opposite (Gontier et al., 2020; Arabshahi et al., 2021; Kalyanpur et al., 2022; Dalvi et al., 2022).
In another recent line of work, chain-of-thought prompting (Nye et al., 2021;
Wei et al., 2022; Wang et al., 2022) encourages LMs to generate intermediate reasoning steps before producing a final answer to a problem.
However, the lack of explicit chaining between the reasoning steps and the final output may compromise the faithfulness of those steps, which have also not yet been evaluated as explanations of model behavior per se. Recent work has explored ways to force these reasoning steps to be more like deductive proofs of the final answer (Creswell & Shanahan, 2022) or instead use generations from a larger language model as silver supervision for a smaller pipeline student model (Eisenstein et al., 2022). In contrast, we aim to develop a method whose rationales (our summarization programs) exactly describe the reasoning process of the overall system and we explicitly evaluate their faithfulness. Specifically, we generate silver programs by our search algorithm that tries to emulate the human summaries and then train models that generate programs which are further executed to obtain final summaries (and evaluated via a simulatability study).
S UMMARIZATION P ROGRAM
We assume that we have a document D
= { D i } di =1 consisting of d sentences and a corresponding abstractive summary S =
{ S i } si =1 consisting of s sentences.
A Summarization Program
P = { T i } si =1 is defined as an (ordered) list of s binary trees where each tree T i
=
( V i , E i ) is a structured representation of the generative process of each summary sentence S i ∈ S .
Fig.shows an example of an SP with two trees for two summary sentences.
The set of nodes V i in each tree consists of single sentences, and the edges E i are labeled with one of the neural modules m ∈ { paraphrase ( · ) , compression ( · ) , fusion ( · , · ) } .
These modules represent operations over sentences wherein compression ( X ) → Y and paraphrase ( X ) → Y are unary operations and fusion ( X , Y )
→ Z is a binary operation.
The leaf nodes in each tree are sentences from the document D i ∈ D , and the root is a summary sentence S i ∈ S .
All other nodes are intermediates sentences generated by executing a neural module (referred to as I1 and I2 in Fig. 1).
An edge from a node u ∈ V i to a node v ∈ V i labeled with the module m means that v is generated by executing m on u .
The summary S is obtained by concatenating the root nodes of the trees in order.
We hypothesize that the generative process of each summary sentence can be captured by composing different neural modules that operate over sentences.
Following prior work on modular approaches to abstractive summarization (Jing & McKeown, 1999; 2000; Lebanoff et al., 2019; 2020b), we define the following three neural modules for building Summarization Programs.
Fusion Module.
Sentence fusion in summarization combines information from multiple sentences (Lebanoff et al., 2020a).
We finetune a BART-large model, which takes two sentences as input and outputs one fused sentence.
Existing sentence fusion datasets either aim to improve discourse connections (Geva et al., 2019) or aim to fuse similar sentences from multiple documents (Brook Weiss et al., 2022).
Instead, we want to fuse two disparate sentences into one sentence, which requires the model to merge related pieces and remove unimportant information.
To obtain training data for such a model, we follow Lebanoff et al.
(2019) to align each summary sentence from CNN/DailyMail with one to many similar and non-redundant document sentences.
As our training data, we only use examples that align one summary sentence with two source sentences.
Compression Module.
The compression module generates a compressed output of a single sentence.
It involves generating a shorter sentence by preserving the essential content and the syntactic structure of the input.
We finetune a BART-large model (Lewis et al., 2020) on a large parallel corpus of uncompressed and compressed sentences (Filippova & Altun, 2013).
Paraphrase Module.
The paraphrase module generates a sentence that involves syntactic transformations or lexical paraphrasing of the input sentence.
We use a publicly available PEGASUSbased (Zhang et al., 2020) paraphrase model from HuggingFace (Wolf et al., 2020).
In practice, we observe paraphrased outputs to frequently involve some compression as well, which we analyze in detail as part of the ‘Neural Module Faithfulness’ evaluation (§6.2).
RQ1: S UMMARIZATION P ROGRAM S EARCH
Our first research question of interest is whether given a document D and a human-written summary
S , we can generate a Summarization Program P that best explains the generative process of the summary (see the left part of Fig. 2).
We achieve this by developing an efficient best-first search method, named SP-S EARCH , outlined in Algorithm 1 in the Appendix.
Conceptually, it is similar to a forward chaining algorithm in logic programming (Russell & Norvig, 2009), in which we start from a set of premises (equivalently, a small set of document sentences) and iteratively apply neural modules on them to generate newer deductions (equivalently, intermediate sentences) until the goal hypothesis (equivalently, the summary) is generated.
SP-S EARCH processes each summary
Pre-trained Modules
SP-Search Algorithm =
[D1, D2, D3, D4, D5, ...]
= [S1, S2]
D
D
I1 fusion fusion D
S1' fusion fusion
D
S2' compression
SP-Model (Extract-and-Build) =
[D1, D2, D3, D4, D5, ...]
= [fusion(<D1>, fusion(<D2>, <D5>))]
[paraphrase(<D3>)]
D
D D
I
S1' fusion fusion fusion fusion
S2' paraphrase
SP-Model (Joint)
MatchSum [D1, D2, D3, D5] = [D1, D2, D3, D4, D5, ...]
D
OR
SP-Search
SP-Model fusion compresssion paraphrase =
[fusion(fusion(<D1>, <D2>), <D5>))] supervision
Top-K [D1, D2, D3, D5] [S1, S2]
Searched Summary: [S1', S2']
Predicted Summary: [S1', S2'] [compression(<D3>)] sentence separately, thereby generating a unique tree for each summary sentence. We optimize the trees for their ROUGE-L (R-L) 4 scores with respect to the gold summary sentences. SP-S EARCH Algorithm. SP-S EARCH operates as follows (see Algorithm 1).
For each summary sentence, it initializes a priority queue, each element ( s 1 , s 2 , m, h ) of which represents a module m , the sentences s 1 and s 2 on which m is defined ( s 2 is empty for ‘compression’ and ‘paraphrase’ modules) and the corresponding tree height h . The initial elements in the queue represent possible operations only on the document sentences. Next, at each step of the search, SP-S EARCH pops an element from the queue and executes the operation over the corresponding operands to generate a new sentence. Each module generates top-5 outputs using beam search and the one with the highest R-L score is chosen.
Using this new sentence, SP-S EARCH creates new elements for the priority queue by considering all potential operations involving the new sentence and other available sentences.
Then, these new elements may be explored in the next steps of the search.
As a best-first search method, SP-S EARCH ranks the elements in the queue with the following scoring function.
[Equation]
The scoring function considers the maximum of the R-L scores with respect to generations s 1 and s 2 such that if the module m is executed, it can possibly lead to a sentence that is of an even higher
R-L score than its children.
Intuitively, it encourages prioritizing elements in the queue that can potentially lead to higher scores.
Whenever a new sentence is generated, its R-L score with the gold summary sentence is computed and accordingly, the best possible root node of the tree is updated.
Upon completion of the search (when the queue becomes empty), the node with the maximum score is chosen as the root node and the tree rooted at that node is the final tree.
Since performing SPS EARCH exhaustively can be prohibitive, it is made efficient through the following design choices.
Design Choices for Efficient SP-S EARCH .
Here we discuss the choices briefly and provide more details in Appendix A. (1) Top-k Document Sentences.
We rank each document sentence by computing ROUGE-1 with the summary and build the SP with only the top-k document sentences as eligible source sentences.
(2) Filtering Queue Elements.
To limit the search space, SP-S EARCH defines certain heuristics for choosing the elements to be added to the queue like a compressed sentence is not compressed again, each document sentence is used at most once in a tree, etc.
(3) Priority Queue and Pruning.
SP-S EARCH maintains a fixed queue size and a ranked list of the elements according to Eq.
1 such that only the top-ranked elements are kept in the queue and the rest are pruned.
(4) Parent has a higher ROUGE than children.
A branch of the search is only expanded further if the new generation (parent) has a higher R-L score than the source sentence(s) (children).
This greedy approach ensures that every reasoning step is a step closer to the summary.
(5) Maximum
Tree Height.
SP-S EARCH chooses a maximum height of the trees.
(6) Batching Module Executions.
SP-S EARCH executes all operations in the queue together by batching at each depth of the search.
RQ2: A BSTRACTIVE S UMMARIZATION VIA S UMMARIZATION P ROGRAMS
Given that we have proposed SP-S EARCH for identifying SPs for human summaries, we now want to leverage the algorithm to generate training data for building SP generation models, as part of RQ2.
In particular, we use SP-S EARCH to identify Summarization Programs for all training samples in the CNN/DailyMail dataset (Hermann et al., 2015).
Our second research question now asks whether these identified SPs can be used as supervision for developing abstractive summarization models via the generation of SPs (see the right part of Fig. 2).
Hence, we define a supervised learning problem f : D → P that generates a Summarization Program P from document D .
Summarization Program Generation Models.
We propose two initial models for our task.
The
SPs are encoded as strings such that they can be generated by a seq2seq generative model.
We first label each sentence in the document with unique identifiers like ‘ < D1 > ’, ‘ < D2 > ’, etc.
The
Summarization Program is then represented using a nested bracketing structure composed of the modules and the sentence identifiers as shown in Fig.
For our first Extract-And-Build Summarization Program Generation Model , we hypothesize that a document can contain a large number of sentences but only a small fraction of those are typically useful for generating a summary.
Given a training corpus of samples ( D k , P ) where D k represents the set of k document sentences on which the SP P is built, we finetune BART such that it can generate a Summarization Program given a small number of document sentences.
During inference, given a document, we first extract a set of relevant sentences using a state-of-the-art extractive summarization model, MatchSum (Zhong et al., 2020), which are then used to generate the program.
While this model simplifies the learning problem by separating out the sentence extraction phase from the program building phase, relevant sentence extraction is an essential first step towards generating good SPs.
Our next Joint Summarization Program Generation Model aims to solve sentence extraction and SP generation as part of a single model.
We finetune BART on the parallel corpus of document and Summarization
Programs and the model learns to directly generate the program given an entire document.
Summary Generation from an SP.
We parse and execute a generated SP through iterative inference over the neural modules to obtain the final summary.
We check for well-formedness of the SP by ensuring that the generated sequence (1) has balanced parentheses, (2) does not contain any out-of-vocabulary token (besides module names, sentence identifiers, and parentheses), and (3) has consistent number of operands for an operation.
During inference, we decode up to top-k SPs using beam search and execute the first well-formed one.
When none of the top-k outputs is well-formed ( < 1% of the samples), we generate the corresponding extractive summary.
E XPERIMENTS
We experiment with two English single-document summarization datasets, CNN/DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018). We discuss CNN/DM results below and point readers to Appendix F for XSum results.
6. RQ1 R ESULTS : C AN SP-S EARCH R EPRESENT THE S UMMARIZATION P ROCESS ?
Our first set of experiments is aimed at answering our first RQ (as discussed in §4) where the goal is to analyze how well the Summarization Programs identified by SP-S EARCH represent the humanwritten summaries.
We consider two variants of SP-S EARCH – (1) SP-S EARCH Top-1 in which each module generates only one output, and (2) SP-S EARCH where each module generates Topoutputs (via beam search) and one with the best R-L is chosen.
Evaluation Metrics.
We use ROUGE to measure similarity between the SP-S EARCH summaries and the human summaries.
We also compute factuality of the SP-S EARCH summaries with respect of the gold summaries using a SOTA factuality metric, QuestEval (Scialom et al., 2021).
Note that while factuality is typically measured against the source document, RQ1 requires us to evaluate how factual the SP-S EARCH summary is to the reference summary (that the SPs are aiming to reproduce).
Experimental Design.
We conduct experiments with 1000 random validation samples.
5 We set the number of initial document sentences ( k ) to 4 , the maximum queue size ( Q ) to 20 , maximum height of the trees ( H ) to 2 and the decoding strategy for each module to beam search with beam size of 5 .
Our choice of hypeparameters is based on a comprehensive study of the trade-off between
ROUGE scores and average search time, as presented in Appendix C. As baselines, we consider the following extractive and abstractive oracles.
(1) Random SP.
For each summary sentence in the gold summary, we randomly sample a tree structure (from those obtained through SP-S EARCH ) and execute it with randomly chosen leaf nodes from the set of top-k sentences.
(2) Top-4 Sentences.
Our next baseline is an extractive summarization system with the top-4 document sentences ranked according to ROUGE-1 scores with the gold summary.
(3) BART-Oracle.
We also compare with two BART-oracle models.
In one, we generate top-10 summaries using beam search.
In another, we sample 10 summaries with multinomial sampling and the summary with the highest ROUGE-L score with the gold summary is chosen.
(4) SP-S EARCH Leaves.
Our final baseline is an SPS EARCH variant where we only consider the leaf nodes (document sentences).
This represents an extractive summary with upto 6 top-4 sentences.
Since the trees are built on top of these nodes, a better ROUGE score for SP-S EARCH will indicate that these leaf nodes or document sentences are being composed in a way such that the resultant summary is more similar to the human summary.
(5) Ablations of Modules.
These are SP-S EARCH variants where each module is removed.
[Table]
As shown in Table [Table] SP-S EARCH generates summaries that better emulate gold summaries by obtaining an R-2 of 40, a significant 7 20 points improvement ( p < 0 . 001 ) over the SP-S EARCH leaves.
The random SP baseline uses the same pre-trained modules as SP-S EARCH but only obtains an R-2 of 15, demonstrating that arbitrary composition of these modules is much less effective than the searched programs. The
BART-oracle models exhibit less diversity between the generations, as reflected by their lower ROUGE scores.
This demonstrates the utility of a tree structure in an SP.
Finally, we observe that each of our neural modules plays a significant role in constructing more accurate SPs.
Removing the fusion module leads to the highest drop in ROUGE which suggests that fusing information from multiple sentences is one of the core operations in human summaries.
SP-S EARCH also obtains the best QuestEval score, even though it does not specifically optimize for it, suggesting that the summaries are not only similar to the human summaries (as measured by ROUGE) but also have high factual overlap with the human summaries.
In summary, SP is an effective way of representing the generative process of abstractive summaries, outperforming random SPs, extractive oracles and unstructured abstractive oracles.
6. RQ1 R ESULTS : N EURAL M ODULE F AITHFULNESS E VALUATION
Higher ROUGE and factuality scores obtained by SP-S EARCH with respect to the reference summaries do not guarantee that SPs provide a faithful interpretation of the generated summaries.
§6.1 specifically evaluates the final summaries but not the intermediate steps used to generate them.
In fact, prior work on Neural Module Networks has shown that faithfulness may not always be guaranteed for neural modules and that incorrect intermediate outputs can still lead to correct final outputs (Subramanian et al., 2020) [Table]
Hence, we also evaluate the faithfulness of the three modules by conducting a human study (with 2 experts having knowledge of NLP) for 20 SPs.
These constitute a total of 107 samples, including 54 fusion, 25 paraphrase, and 28 compression operations.
Table 2 shows the results (with more details in Appendix E).
The diagonal entries in the table demonstrate how often a module performs its intended behavior and the high values of 0.8-0.98 suggest that our modules are highly faithful.
Interestingly, the values in the last column demonstrate that compression and paraphrase modules almost never generate non-factual outputs while our fusion module is more prone to that (about 20%), indicating room for improvement in the fusion module.
6. RQ2 R ESULTS : E VALUATION OF S UMMARIZATION P ROGRAM G ENERATION M ODELS
Our next set of experiments addresses our second RQ (as discussed in §5) on evaluating Summarization Program Generation models in terms of their summary generation capabilities.
Experimental Design.
We compare our SP generation models (Joint and Extract-and-Build) with the following baselines and oracles on the CNN/DailyMail test set.
As baselines, we consider (1) a SOTA Extractive Summarization Model, MatchSum (Zhong et al., 2020) that we also use to extract initial document sentences for our Extract-and-Build SP generation model, (2) SOTA
Abstractive Summarization Models, BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020) , and (3) Random SP Models that randomly sample the number of summary sentences from { 1,2,3,4 } , and for each summary sentence, randomly sample and execute a tree structure with randomly chosen leaf nodes from all or top-k document sentences.
As oracles, we consider the same models introduced in §6.1 for RQ1.
They provide an estimate of the upper bound of SP models.
Evaluation Metrics.
Besides ROUGE, we also evaluate the factuality of the generated summaries (with respect to the source document) using QuestEval that is shown to correlate well with humans.
[Table]
-the-art factuality metric, QuestEval
(Scialom et al., 2021).
Results.
Table 3 shows the results.
We observe that our models obtain R-2 and
R-L scores of 16 andrespectively.
The Extractand-Build model performs slightly better than our
Joint model, possibly due to the former generating programs over a good initial set of document sentences.
Both of our models outperform the Random
SP baselines, demonstrating that they learn useful patterns of which neural modules should act on which document sentences.
Compared to
SOTA abstractive models, our models’ interpretability (as discussed in §6.4) comes at the cost of some drop in performance (about 5 points in R-2).
However, the oracle results suggest that our models provide good starting points for better SP generation models and that there is substantial room for improvement in future work (e.g., the SP-S EARCH Top-1 model obtains an R-2 of 35, leaving a large gap of 19 R-2 points to oracle SPs).
We hope the community will explore and improve upon our SP generation models, as a way to make progress on the important and challenging task of developing interpretable abstractive summarization models.
We observe a similar trend for the QuestEval scores where our models largely outperform the random SP baselines, while showing worse performance than BART and PEGASUS, possibly because of the cascading effect of compositional generation via SPs.
Discussion. BART and PEGASUS optimize for gold summaries while our SP models optimize for SP-Search summaries. The SP models, by themselves, are simple and effective – when the generated summaries are evaluated using SP-Search summaries as targets, we obtain a comparable R-2 of 21 to SOTA models.
This shows that retraining our models on even better oracle programs by incorporating more modules to SP-S EARCH or further enhancing the existing ones can help close the gap to SOTA models. We hope the community will explore and improve upon our methods, as a way to make progress on the challenging task of developing interpretable summarization models.
6. RQ2 R ESULTS : I NTERPRETABILITY E VALUATION OF SP VIA S IMULATION S TUDY
We ask if SPs improve the interpretability of a summarization model.
In particular, we are interested in evaluating model simulatability , a measure of whether a person can predict model behavior on new inputs (Doshi-Velez & Kim, 2017).
Similar to Zhou et al.
(2022), we are specifically interested in model reasoning , as represented by the programs that our models generate and execute.
The primary motivation for predicting model reasoning is that it is what we want to better understand by virtue of model explanations, and simulation studies that focus on predicting final outputs do so only to show that users have a good mental model of model reasoning (Hase & Bansal, 2020).
[Table]
Study Design.
With 3 experts (authors) having prior knowledge in NLP and explainability, we design and carry out a small-scale human study that tests whether presenting humans with Summarization Programs improves their ability to predict SPs on future unseen data.
Human subjects are first provided with some test documents and corresponding model summaries (from our Explain-and-Predict SP model) and are tasked with predicting the model’s Summarization Programs.
Next, in a training phase, they are shown model-generated SPs for a few training samples and are then asked again to perform the same task with the same set of test examples.
The study is conducted with 10 training examples and 31 test examples.
We say that an SP is more representative of the model summary if executing it with our pre-trained modules generates a summary that is closer to the model summary.
This is a measure of program similarity since variance in the SPs may make graph similarity measures inappropriate (e.g., compression followed by paraphrase and vice versa may generate similar outputs).
Results.
The results are presented in Table 4.
In general, high ROUGE scores suggest that, when models are forced to generate summaries via SPs, their reasoning becomes quite predictable a priori.
We also see about 4 points improvement in ROUGE-2, with 61% of samples showing an improved
R-2 score (statistically significant at p < 0 .
001 ), 10% of samples with a drop in R-2, and remaining being ties.
This suggests that SPs are potentially good explanations of model reasoning, such that humans can generalize across model reasoning patterns after being given the explanations.
D ISCUSSION AND C ONCLUSION
We proposed the Summarization Program, a novel framework for interpretable abstractive summarization. We demonstrated its effectiveness by developing SP-S EARCH that identifies summarization programs for human summaries with highly faithful neural modules and SP models that produce summaries from source documents. Two most common forms of errors in the generated SPs include (1) redundant or longer paths in an SP, and (2) fusion module generating non-factual sentences or ignoring one of its source sentences.
Two other notable issues, arising out of the independence assumption of summary sentences, are (1) final summary sentences having overlapping information, and (2) incoherence between consecutive sentences.
One way to improve this is to add a ‘coherence’ module on top of the root nodes before generating the final summary.
We build SPs using sentences as the fundamental content unit (nodes) due to the relative ease of defining and training neural modules on sentences and the availability of large-scale training data.
Summarization may also involve other text manipulation operations that are not fully captured by our modules but our framework allows easy inclusion of other modules.
E THICS S TATEMENT
Despite the recent success of pre-trained language models in abstractive summarization, their lack of explainability still remains a major concern and we hope that Summarization Programs prove to be an important step in bridging that gap.
That said, summarization is inherently a subjective task and existing summarization datasets also significantly vary in terms of their stylistic features like abstractiveness, length, and specificity (Goyal et al., 2022).
Hence, more future work is needed to understand the general applicability of our neural modules and how effective they are in encoding different kinds of summaries.
Broadly put, Summarization Program is a case study for breaking a complex NLP problem down into sub-problems and then solving them through neural modules without having access to intermediate supervision.
We fine-tune language models for building modules and language models can be prone to generating unwanted content (Weidinger et al., 2021). However, since each module is focused on one particular skill, that should help limit the negative impact and provide users with more control, compared to end-to-end models. Summarization Programs are also specifically designed to trace the origin of any toxic or hallucinated content in the generated summaries.
R EPRODUCIBILITY S TATEMENT
To encourage reproducibility, we make our source code publicly available.
The details of our SPS EARCH algorithm are shown in Algorithm 1.
The hyperparameters for RQ1 and RQ2 are discussed in Appendix C and Appendix D respectively.
The CNN/DailyMail and XSum datasets are also publicly available at https://huggingface.co/datasets/cnn_dailymail and https://huggingface.co/datasets/xsum respectively. A CKNOWLEDGEMENTS
We thank the reviewers for their valuable feedback.
We also thank Archiki Prasad and Prateek Yadav for their help with neural module faithfulness annotations, and David Wan for help with factuality evaluation.
This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute
DRL-2112635, Google Ph.D. Fellowship, and Bloomberg Data Science Ph.D. Fellowship
A D ESIGN C HOICES FOR E FFICIENT SP-S EARCH (C ONTINUED FROM §4)
SP-S EARCH is outlined in Algorithm 1.
It is made efficient through important design choices as discussed below.
For clarity, we do not show all of these choices in the Algorithm outline.
Top-k Document Sentences.
The search space grows exponentially at each depth (because of the fusion operation), but one way to limit the growth is to ensure that SP-S EARCH starts with a small number of document sentences.
A summary is typically constructed by using information from only a small number of sentences from the document.
Hence, we rank each document sentence by computing the ROUGE-1 score with the summary and build the Summarization Program with only the top-k document sentences (‘k’ being a small number) as eligible source sentences.
Filtering Queue Elements.
The search space is also dependent on how many elements are added to the queue at each step and how many of those are expanded further.
Note that whenever a new sentence is generated via a module, it can potentially fuse with all previous generations to create new queue elements.
Since doing this exhaustively increases the search space, SP-S EARCH defines certain heuristics for choosing the elements that will be added to the queue: (1) a sentence that has been compressed once is not compressed again, (2) a sentence that has been paraphrased once is not paraphrased again, (3) each document sentence is used at most once in a tree, (4) two sentences are not fused if they are intermediate generations from the same sentence, and (5) since fusion is not a symmetric operation and can lead to different generations based on the order, sentence fusion happens keeping the temporal order of the sentences in the document intact.
Priority Queue and Pruning.
SP-S EARCH performs an additional step of pruning for the elements that are added to the queue.
It maintains a fixed queue size and while expanding the elements in the queue in a best-first manner, it maintains a ranked list of the elements according to Eq.
1 such that only the top-ranked elements are kept in the queue and the rest are pruned.
Parent has a higher ROUGE than children.
Each module generates multiple outputs through beam search and SP-S EARCH ensures that the best output has a higher R-L score than the nodes on which the module is defined.
If this is not the case, the corresponding branch of the search is not expanded further.
This constraint generalizes to the property that every node in a Summarization
Program will have a higher R-L score (with respect to the summary sentence) than all other nodes in the subtree rooted at that node.
Conceptually, this greedy approach ensures that every reasoning step in a Summarization Program is a step closer to the summary (according to a scoring function).
Maximum Tree Height.
SP-S EARCH chooses a maximum height of the trees, beyond which the nodes are not expanded further during the search.
Batching Module Executions.
Instead of executing each module separately,
which requires a forward pass over a neural model and can be time-consuming, SP-S EARCH executes all operations in the queue together by batching at each depth of the search. visited ← compression queue ← queue ∪ { ϕ M height
} ▷ Add paraphrase items.
forall prevGen ∈ visited do queue ← queue ∪ { newGen prevGen
M f height ) } ▷ Add fusion items. P return P B A NALYSIS OF SP-S EARCH S UMMARIZATION P ROGRAMS F OR CNN/DM
We observe that 68% of the summaries in CNN/DM have 3 or 4 summary sentences and, equivalently, the corresponding Summarization Programs have 3 or 4 trees.
Note that while we initialize
SP-S EARCH with the Top-4 document sentences, a Summarization Program may choose to ignore some of these sentences if including them does not lead to higher ROUGE scores.
Upon analysis, we find that 73% of the programs are constructed using all four initial sentences, 23% are constructed with three sentences and 3% have two sentences.
We also note that the trees can have any height up to the maximum defined height of 2.
A tree of height 0 represents a singleton node with a single document sentence.
Thus, a Summarization Program with only singleton nodes represents an extractive summary.
Overall, we observe that our trees have as many as 20 different structures.
Tableshows the top 10 tree structures in our corpus.
As an example, a tree structure of “compression ( [Table] [Table] fusion ( · , · ) )” represents a tree of height 2 in which two document sentences are first fused and then the resultant sentence is compressed.
C SP-S EARCH H YPERPARAMETERS
In order to analyze the effect of different hyperparameters in SP-S EARCH , we compare the trade-off between ROUGE 8 scores and average search time per sample (on a single RTX 2080 Ti GPU) by varying the number of initial document sentences ( k ), the queue size ( Q ), decoding strategy ( D ) and the maximum height of the trees ( H ) in Table 6.
We observe that increasing the number of document sentences k beyond 4 leads to some improvements in ROUGE scores.
However, this comes at the cost of increased search time.
Increasing the queue size ( Q ) also leads to minor improvements but again suffers from increased search time.
While increasing the maximum tree height ( H ) to 3 results in better ROUGE scores, on inspection, we observe that it happens primarily due to low-quality fusion of two arbitrary sentences that may not always lead to better programs.
Finally, beam search performs better than greedy decoding, and leveraging the best of the top-5 generations from each module improves the results further with almost no increase in search time.
Overall, at a moderate search time of less than 30 seconds/sample on a single RTX 2080 Ti GPU, SP-S EARCH obtains a 20 point increase in R-2 compared to our extractive baseline with leaf nodes (document sentences).
SP-S EARCH builds Summarization Programs by optimizing for ROUGE-L because of its better overall performance across all ROUGE metrics.
As shown in Table 7, optimizing instead for Rand R-2 leads to slightly better R-1 and R-2 respectively but overall R-L performs better.
In general, [Table] [Table] we find that the different ROUGE metrics may not correlate well with each other when optimizing for one of these.
SP-S EARCH uses ROUGE-1 scores to select the candidate sentences for building the Summarization Programs.
We also experiment with other ROUGE metrics in Table 8 and observe that our algorithm is fairly robust to the choice of metric.
R-2 obtains slightly better results than R-1 and R-L but overall, all metrics still obtain sufficiently high oracle results of 40-41 points in R-2 through SPSearch, independent of the selection metric. D SP M ODEL H YPERPARAMETERS
We build our models on top of the HuggingFace transformers library (Wolf et al., 2020).
All models are trained for 40000 steps with a batch size of 16 , learning rate of 3 ∗ 10 − 5 and warmup steps of 500 .
We set the maximum input length to 512 and maximum generation length to 100 . During inference, we generate up to Top-10 Summarization Programs with beam search and output the first well-formed program. We also set the minimum generation length to 10 to prevent the model from generating too short sequences and repetition penalty to 2 . Program execution from the SPs is performed with the same set of hyperparamaters for each module as used during SP-S EARCH .
E RQ1 R ESULTS : N EURAL M ODULE F AITHFULNESS E VALUATION ON CNN/DM (C ONTINUED FROM §6.1)
We discuss the faithfulness evaluation of our modules below.
Study Design.
We would attribute each sentence to exactly one of the modules, but since our modules are generative, we do not know if each module performs only and exactly its named function.
For example, fusing two sentences into a fluent sentence may in practice also involve some paraphrasing or compression.
Hence, we evaluate module faithfulness by analyzing how often a module m i performs the role of a module m j .
Two expert annotators, with knowledge of NLP, annotate each intermediate generation (from a module) and assign a binary label against each of the modules that could lead to that output.
Additionally, the annotators also label each generation for whether it contains non-factual content (that cannot be verified from the sentence(s) used to generate it).
The study is conducted with 20 Summarization Programs, consisting of a total of 107 samples, withfusion, 25 paraphrase, and 28 compression operations.
Results.
Table 9 shows the results.
The diagonal entries in the table demonstrate how often a module performs its intended behavior and the high values of 0.8-0.98 suggest that our modules [Table] are highly faithful.
In about 20% of the cases, a fusion operation may not involve any fusion.
We also observe that some form of compression frequently happens as part of the paraphrasing and fusion modules, as shown in the first column of the table.
Similarly, the fusion outputs also tend to involve some compression and paraphrasing operations.
Interestingly, the values in the last column of the table demonstrate that compression and paraphrase modules almost never generate non-factual outputs while our fusion module is more prone to that (about 20%), indicating room for improvement in the fusion module.
F R ESULTS ON XS UM D ATASET (C ONTINUED FROM §6)
To further test the generalizability of Summarization Programs, we conduct experiments on another single document summarization dataset, XSum (Narayan et al., 2018), a highly abstractive dataset consisting of single-sentence summaries.
F. RQ1 R ESULTS : C AN SP-S EARCH R EPRESENT THE S UMMARIZATION P ROCESS ?
As part of answering RQ1, we keep the paraphrase and compression modules unaltered and only retrain the fusion module for XSum, with the training data obtained using the same heuristics as used for CNN/DM (Lebanoff et al., 2019).
We also use the same hyperparameters for SP-S EARCH as those used for CNN/DM and compare all methods on 1000 randomly chosen validation samples of XSum.
As shown in Table 10, SP-S EARCH obtains an R-2 of 29.60 and an R-L of 48.48, outperforming all baseline methods by a significant margin.
Unlike CNN/DM, extractive baselines like ‘Lead 1’, ‘SP-S EARCH Leaves’, and ‘Top-1 Sentence’ do not perform well for XSum, while the abstractive baseline, BART-Oracle does significantly better.
The SP-S EARCH results are also lower than in CNN/DM because of the highly abstractive nature of the dataset and the relative difficulty in emulating reference summaries.
F. RQ2 R ESULTS : E VALUATION O F S UMMARIZATION P ROGRAM G ENERATION M ODELS
Next, for RQ2, we train a Joint SP generation model that generates the program directly from the document.
In Table 11, we compare our model with BART and PEGASUS and a random SP generation baseline.
While our model outperforms the random SP baseline, there’s significant room for improvement compared to state-of-the-art methods, as indicated by the 16 point gap in Rcompared to the SP-S EARCH oracle.
G E XAMPLES OF S UMMARIZATION P ROGRAMS
Figures 3, 4, 5, and 6 show some examples of Summarization Programs identified by SP-S EARCH for human-written summaries.
Figures 7, 8, and 9 show examples of Summarization Programs and corresponding summaries generated by our Extract-and-Build SP generation model.
For compactness, we show the Summarization Programs by merging the common nodes between the trees.
[Table] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
, have
[Table]
four drones to help control crowds
[Table]
The unmanned aerial vehicles are being fitted with cameras and
[Table]
subdue
[Table]
Some
[Table]
have
[Table]
why police are
[Table]
[Figure] [Figure]
Predicted Summary: Manning appears to have joined Twitter this week.
Manning said she suffers from gender dysphoria.
In August 2013, Manning said she wanted to transition from male to female after being sentenced to 35 years in prison.
[Figure]
[Table]
year
[Table]
Manning said she
[Figure]
H S IMULATABILITY I NTERFACE OF S UMMARIZATION P ROGRAMS
In Figure 10, we show the interface for our simulatability study in which annotators are asked to write Summarization Pograms for model-generated summaries.