Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution (2021)

Abstract: Despite the prominence of neural abstractive summarization models, we know little about how they actually form summaries and how to understand where their decisions come from. We propose a two-step method to interpret summarization model decisions. We first analyze the model’s behavior by ablating the full model to categorize each decoder decision into one of several generation modes: roughly, is the model behaving like a language model, is it relying heavily on the input, or is it somewhere in between? After isolating decisions that do depend on the input, we explore interpreting these decisions using several different attribution methods. We compare these techniques based on their ability to select content and reconstruct the model’s predicted token from perturbations of the input, thus revealing whether highlighted attributions are truly important for the generation of the next token. While this machinery can be broadly useful even beyond summarization, we specifically demonstrate its capability to identify phrases the summarization model has memorized and determine where in the training pipeline this memorization happened, as well as study complex generation phenomena like sentence fusion on a per-instance basis.

## introduction transformer-based neural summarization models @xcite @xcite , especially pretrained abstractive models like bart @xcite and pegasus @xcite , have made great strides in recent years. these models demonstrate exciting new capabilities in terms of abstraction, but little is known about how these models work. in particular, do token generation decisions leverage the source text, and if so, which parts? or do these decisions arise based primarily on knowledge from the language model @xcite , learned during pre-training or fine-tuning? having tools to analyze these models is crucial to identifying and forestalling problems in generation, such as toxicity @xcite or factual errors @xcite @xcite . although interpreting classification models for nlp has been widely studied from perspectives like feature attribution @xcite and influence functions @xcite , summarization specifically introduces some additional elements that make these techniques hard to apply directly. first, summarization models make sequential decisions from a very large state space. second, encoder-decoder models have a special structure, featuring a complex interaction of decoder-side and encoder-side computation to select the next word. third, pre-trained lms blur the distinction between relying on implicit prior knowledge or explicit instance-dependent input. this paper aims to more fully interpret the stepwise prediction decisions of neural abstractive summarization models. 1 first, we roughly bucket generation decisions into one of several modes of generation. after confirming that the models we use are robust to seeing partial inputs, we can probe the model by predicting next words with various model ablations: a basic language model with no input (lm ∅ ), a summarization model with no input (s ∅ ), with part of the document as input (s part ), and with the full document as input (s full ). these ablations tell us when the decision is context-independent (generated in an lm-like way), when it is heavily context-dependent (generated from the context), and more. we map these regions in figure 2 and can use these maps to coarsely analyze model behavior. for example, 17.6% of the decisions on figure 1 : our two-stage ablation-attribution framework. first, we compare a decoder-only language model (not fine-tuned on summarization task, and not conditioned on the input article) and a full summarization model. they are colored in gray and orange respectively. the the higher the difference, the more heavily model depends on the input context. for those context-dependent decisions, we conduct content attribution to find the relevant supporting content with methods like integrated gradient or occlusion. xsum are in the lower-left corner (lm-like), which means they do not rely much on the input context. second, we focus on more fine-grained attribution of decisions that arise when the model does rely heavily on the source document. we carefully examine interpretations based on several prior techniques, including occlusion @xcite , attention, integrated gradients @xcite , and input gradients @xcite . in order to evaluate and compare these methods, we propose a comprehensive evaluation based on presenting counterfactual, partial inputs to quantitatively assess these models' performance with different subsets of the input data. our two-stage analysis framework allows us to (1) understand how each individual decision depends on context and prior knowledge (sec 3), (2) find suspicious cases of memorization and bias (sec 4), (3) locate the source evidence for context dependent generation (sec 5). the framework can be used to understand more complex decisions like sentence fusion (sec 6). ## ablation: mapping model behavior based on assumption 1, we can take a first step towards understanding these models based on the partial models described in section 2.3. previous work @xcite has studied model behavior based on externally-visible properties of the model's generation, such as identifying novel words, differentiating copy and generation, and prediction confidence, which provides some insight about model's behavior @xcite . however, these focus more on shallow comparison of the input document, the generated summary, and the reference summary, and do not focus as strongly on the model. we propose a new way of mapping the prediction space, with maps 3 for xsum and cnn/dm shown in figure 2 . each point in the map is a single subword token being generated by the decoder on the development set at inference time; that is, each point corresponds to a single invocation of the model. this analysis does not depend on the reference summary at all. the x-axis of the map shows the distance between lm ∅ and s full , using the metric defined in section 2.4 which ranges from 0 to 2. the y-axis shows the distance between s ∅ and s full . other choices of partial models for the axes are possible (or more axes), but we believe these show two important factors. the x-axis captures how much the generic pre-trained language model agrees with the full model's predictions. the y-axis cap3 while our axes are very different here, our mapping concept loosely follows that of @xcite . tures how much the decoder-only summarization model agrees with the full model's predictions. the histogram on the sides of the map show counts along with each vertical or horizontal slice. ## attribution as shown in here, we focus specifically on identifying which aspects of the input are important for cases where the input does influence the decision heavily using attribution methods. each of the methods we explore scores each word w i in the input document with a score α i . the score can be a normalized distribution, or a probability value ranging from 0 to 1. for each method, we rank the tokens in descending order by score. to confirm that the tokens highlighted are meaningfully used by the model when making its predictions, we propose an evaluation protocol based on a range of counterfactual modifications of the input document, taking care to make these compatible with the nature of subword tokenization. ## case study: sentence fusion we now present a case study of the sort of analysis that can be undertaken using our two-stage interpretation method. we conduct an analysis driven by sentence fusion, a particular class of ctx-hd cases. sentence fusion is an exciting capability of abstractive models that has been studied previously @xcite @xcite . we broadly identify cases of cross-sentence information fusion by first finding cases in ctx-hd where the max(p sent ) < 0.5, but two sentences combined enable the model to predict the word. we search over all m 2 combinations of sentences (m is the total number of sentences) and run the s part model on each pair of sentences. we identify 16.7% and 6.0% of cases in cnn/dm and xsum, respectively, where conditioning on a pair of sentences increases the probability of the model's generation by at least 0.5 over any sentence in isolation. in the majority of the "fusion" cases we investigated actually reflect content selection at the beginning of the generation. other cases we observe fall more cleanly into classic sentence fusion or draw on coreference resolution. ## related work model interpretability for nlp has been intensively studied in the past few years @xcite @xcite @xcite @xcite . however, many of these techniques are tailored to classification tasks like sentiment. for post-hoc interpretation of generation, most work has studied machine translation @xcite @xcite . @xcite focus on evaluating explanations by finding surrogate models that are similar to the base mt model; this is similar to our evaluation approach in section 5, but involves an extra distillation step. compared to @xcite , we are more interested in highlighting how and why changes in the source article will change the summary (counterfactual explanations). to analyze summarization more broadly, @xcite provides a descriptive analysis about models via uncertainty. previous work @xcite @xcite has conducted comprehensive examination of the limitations of summarization models. filippova (2020) ablates model input to control the degree of hallucination. @xcite improves the training of mt by comparing the prediction of lm and mt model. finally, this work has focused chiefly on abstractive summarization models. we believe interpreting extractive @xcite or compressive @xcite @xcite models would be worthwhile to explore and could leverage similar attribution techniques, although ablation does not apply as discussed here. ## recommendations & conclusion we recommend a few methodological takeaways that can generalize to other conditional generation problems as well. first, use ablation to analyze generation models. while removing the source forms inputs not strictly on the data manifold, ablation was remarkably easy, robust, and informative in our analysis. constructing our maps only requires querying three models with no retraining required. second, to understand an individual decision, use feature attribution methods on the source only. including the target context often muddies the interpretation since recent words are always relevant, but looking at attributions over the source and target together doesn't accurately convey the model's decision-making process. finally, to probe attributions more deeply, consider adding or removing various sets of tokens. the choice of counterfactuals to explain is an illposed problem, but we view the set used here as realistic for this setting @xcite . taken together, our two-step framework allows us to identify generation modes and attribute generation decisions to the input document. our techniques shed light on possible sources of bias and can be used to explore phenomena such as sentence fusion. we believe these pave the way for future studies of targeted phenomena, including fusion, robustness, and bias in text generation, through the lens of these interpretation techniques.