Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths (2020)

Abstract: Abstractive summarization at controllable lengths is a challenging task in natural language processing. It is even more challenging for domains where limited training data is available or scenarios in which the length of the summary is not known beforehand. At the same time, when it comes to trusting machine-generated summaries, explaining how a summary was constructed in human-understandable terms may be critical. We propose Multi-level Summarizer (MLS), a supervised method to construct abstractive summaries of a text document at controllable lengths. The key enabler of our method is an interpretable multi-headed attention mechanism that computes attention distribution over an input document using an array of timestep independent semantic kernels. Each kernel optimizes a human-interpretable syntactic or semantic property. Exhaustive experiments on two low-resource datasets in English show that MLS outperforms strong baselines by up to 14.70% in the METEOR score. Human evaluation of the summaries also suggests that they capture the key concepts of the document at various length-budgets.

## introduction great progress has been made in recent years on abstractive summarization of text documents. among existing works, sequence-to-sequence networks with attention @xcite have been one of the clear front-runners. being able to constrain the length of a summary while preserving its desirable properties has many real-world applications. one such application is content optimization for variable screen-sizes. online content creators such as news portals, blogs, and advertisement agencies with audiences on multiple platforms customize their content based on display-area for best experience. however, there has not been much work on summarization at controllable lengths until recently. high variance in screen-sizes often require extensive human supervision to perform these modifications. as most sequence-to-sequence networks @xcite do not enforce the length of a summary, for scenarios as mentioned above, one may need to employ an ensemble of networks to cover all possible lengths. there are two major challenges in following this approach for realworld applications. first, training sequence-to-sequence networks is a resource-intensive task @xcite . to train a network for generating summaries budgeted at length b, we need a parallel corpus of text documents and their gold-standard summaries at length b. constructing a large enough corpus with summaries budgeted at b, ∀@xmath0, 1) may not be possible and/or cost-efficient for a number of domains. this is one of the main reasons why most existing works on abstractive summarization evaluate their model on large-scale news corpus datasets @xcite , leaving out a number of important but low-resource domains @xcite where the number of available training documents is limited. second, the range of possible lengthbudgets r(b) may not always be known beforehand. in many scenarios, it can be known as late as during run-time. therefore, we formalize the summarization task addressed in this paper as follows. problem definition: given a document s of length n (tokens) and a maximum token budget of b, we aim to construct an abstractive summary s b that satisfies the following conditions, c1: information input text police are hunting a man aged between 50 and 60 suspected of robbing a bank in broad daylight and running off with £3,000 in cash. the robbery took place at 12.30pm at a lloyds bank branch in fairwater, cardiff, police said. detectives have issued cctv images of the suspect, who is 50 to 60, 5ft 9in to 6ft and was wearing black clothing. the white male suspect, who has greying black hair and wore glasses, was captured on camera inside the bank. detectives said no one was injured during the robbery and they were 'confident' the public would be able to identify the suspect. detective sergeant andy miles, from fairwater cid, said: 'inquiries are continuing to identify the culprit. the cctv is clear and i am confident that members of the public will know his identity...'. @xcite summary at compression @xmath1,000 in cash. the robbery took place at 12.30pm at a lloyds bank branch in fairwater, cardiff, police said. the white male suspect, who has greying black hair and wore glasses, was captured on camera inside the bank. detectives have issued cctv images of the suspect, who is 50 to 60, 5ft 9in to 6ft and was wearing black clothing. detective sergeant andy miles, from fairwater cid, said: 'inquiries are continuing to identify the culprit. ## proposed methodology mls constructs a length-constrained summary of a document in two steps. first, it derives a prototype summary s p from the document, covering its major concepts. then, it expands or shortens it, depending on the length-budget to create the final summary. we employ a pair of encoder-decoder networks at both steps. for the first step, we extend the pg-network @xcite . we develop a length-aware encoderdecoder network for the second step. we describe both steps in greater detail in the following sections. ## conclusion we have proposed mls, a supervised approach to construct abstractive summaries at controllable lengths. following an extract-then-compress paradigm, we develop the pointer-magnifier networka length-aware, encoder-decoder network that constructs length-constrained summaries by shortening or expanding a prototype summary inferred from the document. the key enabler of this network is an array of semantic kernels with clearly defined human-interpretable syntactic/semantic roles in constructing the summary given a budget-length. we train our network on limited training samples from two cross-domain datasets. experiments show that the summaries constructed by mls are coherent and reflectively capture the main concepts of the document. our human evaluation study also suggest the same. in the future, we would like to extend our work to construct task-driven summaries for interactive question answering tasks. personalizing a summary based on user's past interaction model is another exciting direction of future work.