Attention Head Masking for Inference Time Content Selection in Abstractive Summarization (2021)

Abstract: How can we effectively inform content selection in Transformer-based abstractive summarization models? In this work, we present a simple-yet-effective attention head masking technique, which is applied on encoder-decoder attentions to pinpoint salient content at inference time. Using attention head masking, we are able to reveal the relation between encoder-decoder attentions and content selection behaviors of summarization models. We then demonstrate its effectiveness on three document summarization datasets based on both in-domain and cross-domain settings. Importantly, our models outperform prior state-of-the-art models on CNN/Daily Mail and New York Times datasets. Moreover, our inference-time masking technique is also data-efficient, requiring only 20% of the training samples to outperform BART fine-tuned on the full CNN/DailyMail dataset.

## introduction large pre-trained transformers have achieved stateof-the-art results on various summarization datasets with a fine-tuning phase to streamline the summarization pipeline @xcite ). yet, it is still unclear how one can use large models more effectively for abstractive summarization . for example, prior work shows that informing content selection via attention weight updating in recurrent neural networks can further boost summarizer performance @xcite . however, with multi-heads attentions at all layers in transformers @xcite , highlighting salient content becomes non-trivial. in this work, we propose an inference-time attention head masking mechanism that works on encoder-decoder attentions to underscore salient content from the source and improve the quality of abstractive summaries. based on this mechanism, we first demonstrate the relation between encoderdecoder attentions and content selection behaviors, on three summarization datasets of cnn/dailymail (cnn/dm), new york times (nyt), and xsum. second, we study whether multiple heads at the same layer collectively guide the summarization. partial masking is found to be most effective, indicating a strong collaborative effect and the importance of head selection. based on these observations, we evaluate attention head masking on summarization benchmarks with salience labels provided by externally trained content selectors. on all three datasets, our model consistently outperforms fine-tuned bart @xcite and several top performing transformer-based abstractive summarization models @xcite . summaries generated by our model are also considered to have better informativeness by human judges. moreover, we illustrate that attention head masking is data-efficient: on cnn/dm, bart fine-tuned on less than 20% of the training data outperforms a version trained on the full set. finally, we show that our method is effective under a cross-domain setting. with a content selector trained on nyt, bart fine-tuned on cnn/dm gains more than three points of rouge scores when tested on nyt articles. foot_0 2 related work large pre-trained models for summarization. many recent advancements in text summarization have been achieved by large pre-trained language models @xcite @xcite . in particular, bart has demonstrated impressive performance on summarization, and is used as the base model in this work. nonetheless, all prior attempts take pre-trained models as is and conduct fine-tuning on target datasets, without knowing if it is the most effective usage. in contrast, we bring insights into the relation between attentions and content selection via masking operations to further improve summarization performance. ## attention head masking we adopt large pre-trained sequence-to-sequence transformer models (bart, specifically) for abstractive summarization. transformer is built with multi-head attentions. attentions are computed per step based on a query q along with the key and value matrices, k and v: ## encoder-decoder attentions and content selection in this section, we first probe into the content selection behavior of each single head ( § 4.1), and then study the synergism among heads at the same layer ( § 4.2). in § 4.3, we analyze the attentions' focus. our analysis is conducted on cnn/dm @xcite , nyt @xcite , and xsum @xcite . we follow @xcite for data preprocessing and train/validation/test splits on cnn/dm and xsum, and adopt the setups in @xcite for nyt, except that we keep entities and numbers. the number of samples in training, validation, and test set are: 287,188, 13,367 and 11,490 for cnn/dm; 588,909, 32,716 and 32,703 for nyt; 204,045, 11,332 and 11,334 for xsum. for experiments in this section, we create an analysis set of 1,000 random samples from the validation split of each dataset to reduce computational cost. ## summarization results with attention head masking in this section, we show how to leverage attention head masking and a content selector to improve summary informativeness on three datasets. we first train a binary sequence tagger for each dataset to label salient tokens in the source, used for system masking for attention heads. our sequence tagger is a roberta @xcite encoder followed by a double layer multilayer perceptron (mlp) with a hyperbolic tangent activation function in between. to obtain the probability for each token, the mlp output is further fed into a sigmoid activation function. details for training and decoding are in appendix a. the decision boundary for the sequence tagger is selected according to the f1 score calculated between the predicted tags and the ground-truth labels on the validation set. we search for the best decision boundary from 0.1 to 0.4, with a step size of 0.01. the final decision boundaries used for taggers trained on cnn/dm, nyt, xsum are 0.20, 0.24, and 0.18, achieving rouge-1 f1 of 43.70, 44.10, and 31.56, respectively. to select which heads at which layers to mask, we employ a greedy selection strategy. on the analysis set, we gradually apply system masking on four heads with most rouge improvement according to the study in § 4.1, and we select the heads that achieve the highest sum of rouge-1 f1 and rouge-2 f1. we apply four heads each time to reduce computational cost of hyperparameter searching. heads selected for each dataset are in appendix b. in-domain results. for human evaluation, we hire three fluent english speakers to rate 50 pairs of summaries generated with and without attention head masking based on bart for informativeness and faithfulness. informativeness measures how well the summary captures salient content from the article, while faithfulness indicates whether the summary correctly reflects the content in the source article. the annotators are asked to determine if attention head masking improves any of the two aspects. as shown in ## conclusion we propose attention head masking that constrains encoder-decoder attention heads to attend to salient tokens, to inform content selection in abstractive summarization. with this technique, we first demonstrate the relation between encoder-decoder attentions and content selection behaviors. with system masks predicted by external content selectors, we show that attention head masking can consistently improve rouge scores over competitive summarization models on three benchmarks. summaries generated with attention head masking are also preferred by human judges more frequently. additional experiments demonstrate that our method is more data-efficient and effective on both in-domain and cross-domain settings.