Length Control in Abstractive Summarization by Pretraining Information Selection (2022)

Abstract: Previous length-controllable summarization models mostly control lengths at the decoding stage, whereas the encoding or the selection of information from the source document is not sensitive to the designed length. They also tend to generate summaries as long as those in the training data. In this paper, we propose a length-aware attention mechanism (LAAM) to adapt the encoding of the source based on the desired length. Our approach works by training LAAM on a summary length balanced dataset built from the original training data, and then fine-tuning as usual. Results show that this approach is effective in generating high-quality summaries with desired lengths and even those short lengths never seen in the original training set.

## introduction abstractive summarization @xcite @xcite @xcite @xcite aims at reproducing the semantics and topics of the original text in a concise and fluent summary by paraphrasing. in order to display the summary on different mobile devices or websites with space limitations, we have to produce summaries in different lengths. length-controllable summarization is a multi-objective optimization problem, including generating complete summaries within desired lengths and selecting proper information to summarize based on desired lengths. the existing length-controllable summarization based on encoder-decoder models can be divided into two categories: (1) early-stop during decoding and (2) information selection before encoding. early-stop during decoding methods @xcite @xcite source document ... iranians erupted in celebration as young people waved flags from their sunroofs , blasted music from stereos and chatted online with the hashtag #irantalks . the excitement came after a breakthrough nuclear deal with the united states and other world powers ... ## approach in this section, we first introduce the lengthcontrollable summarization (lcs) problem, then introduce the length-aware attention mechanism (laam), which attends the existing transformer seq2seq models, and finally explain how to create a length-balanced dataset (lbd) for pretraining. ## evaluation we first introduce the datasets and the experimental setup. we design two experiments, general length control and zero-shot length control, to compare our approach with baselines. foot_0 general length control experiment trains and tests the models on the entire original dataset. zero-shot length control experiment tests the model on a subset of the test set whose summary lengths fall within a certain range, and trains the model on training data with summary lengths outside this range. in each of the two experiments, we evaluate methods' ability to do length control and information selection. ## related work previously, most length-controllable approaches in abstractive summarization focused on stoping decoding at a particular time. ad-hoc @xcite generated the eos token by assigning a score of -∞ to the tokens in vocabulary and generated a fixed number of words. lenemb and leninit @xcite input length embeddings to decoder respectively. bian et al. (2019) took lenemb and leninit as an agent and adjusted the reward incorporating with the desired length. lc @xcite added the desired length into the first layer of cnn encoder. golc @xcite optimized lenemb and lc by formalizing loss with an overlength penalty. @xcite predefined some special markers to denote different length ranges and prepended the input with such markers during training and test-ing. @xcite extended the sinusoidal positional encoding @xcite to take account of stepwise remaining length. lenatten @xcite added a length attention unit to exploit proper length information based on the stepwise remaining length. other length-controllable approaches decided the content to be summarized by length-aware intermediate summaries. lpas @xcite extracted a word sequence with the desired length from source document and generated summary by a non-length-controllable model with document and extracted summary as input. mls @xcite generated a general summary and then input it to a length-controllable model. compared with previous methods, our approach can effectively control the length of generated summaries by pretraining the length-controllable information selection model on length-balanced dataset. meanwhile, it can generate summaries with length approximate to the desired length in zero-shot controlling length problem. recently, the approaches fine-tune the pretrained transformer seq2seq models @xcite @xcite on summarization datasets. they achieve outstanding performances on summarization tasks. our approach is applied to transformer seq2seq model, which is orthogonal to above pretrained transformer models and can be added to them. ## conclusion we present a novel approach to produce summaries in desired length that are fluent and coherent. this approach pretrains a transformer seq2seq model whose cross attention between input and output are re-normalized accordingly to the length requirement. the pretraining is done over synthetic summarization data extracted from the original training set but with summary lengths evenly distributed. our results show that the framework achieves a good balance between information selection from input documents and length control when producing summaries.