Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021)

Abstract: Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents, outperforming their vanilla counterparts on ROUGE and multiple faithfulness measures. We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-k or nucleus sampling-based decoding methods.

## introduction document summarization -producing the shorter version of a document while preserving salient information @xcite -is challenging even for humans. today, systems can generate summaries with a high level of fluency and coherence. this is due to recent advances such as sequence-to-sequence architectures (seq2seq) with attention and copy mechanism @xcite @xcite , fully attention-based transformer architectures @xcite , and large pretrained language models @xcite a gold: australia has expelled an israeli diplomat saying israel was behind the forging of australian passports linked to the murder of a hamas operative in dubai. pegasus: australia has expelled an israeli diplomat after concluding that forged australian passports used in the killing of a hamas militant in dubai were issued by israel. our pegfame model: the australian government has expelled an israeli diplomat over the use of forged australian passports in the killing of a hamas militant in dubai. b pegasus with top-k sampling israel has summoned the australian ambassador to complain after the australian government said forged passports used in the killing of a hamas operative in dubai belonged to netanyahu's foreign ministry. the australian government has ordered israel to withdraw an officer over the use of forged australian passports used by the 2013 murder of a lebanese opposition figure in dubai. pegasus with nucleus sampling israel hasracuse withdrawn an envoy after the australian government said it concluded that israeli agents used forged passports used to kill a dubai bendigo businessman. the australian government has recalled an israeli diplomat over accusation that fake australian passports used 436 kilometres (300 miles) from canberra in the death of a hamas militant were stolen by israeli agents. ## summarization with focus attention given an input document x 1:n , we aim to generate its summary y 1:m , where n and m are input and output sequence lengths. we address this prob- transformer-based seq2seq model the encoder uses bert transformer layers with multiheaded self-attention to encode x to a vector sequence @xmath0, . . . , x n , with x @xmath1, where h is the size of hidden representation. the decoder uses an identical architecture, except that at decoding step t, layer l adds a conditional representation y l @xmath2:t-1 = y l-1 1 , . . . , y l-1 t-1 generated so far through self-attention and by attending to the input contextual representation x through encoderdecoder attention. the probability of predicting the next token y t from a vocabulary v is: ## experimental setup in this section we present our experimental setup to assess the ability of our fame models to generate faithful summaries and to demonstrate that focus sampling is more effective in generating diverse and faithful summaries than other sampling-based decoding methods. ## conclusion we introduced fame, a new attention mechanism which dynamically biases the decoder to proactively generate tokens that are topically similar to the input. fame enhances the faithfulness of existing state-of-the-art abstract summarization models while improving their overall rouge scores. finally, our newly introduced focus sampling technique is a better alternative to top-k or nucleus sampling to generate diverse set of faithful summaries.