A Decoding Algorithm for Length-Control Summarization Based on Directed Acyclic Transformers (2024)

Abstract: Length-control summarization aims to condense long texts into a short one within a certain length limit. Previous approaches often use autoregressive (AR) models and treat the length requirement as a soft constraint, which may not always be satisfied. In this study, we propose a novel length-control decoding algorithm based on the directed acyclic Transformer (DAT). Our approach allows for multiple plausible sequence fragments and predicts a path to connect them. In addition, we propose a Sequence Maximum a Posteriori (Seq-MAP) decoding algorithm that marginalizes different possible paths and finds the most probable summary satisfying the length budget. Our algorithm is based on beam search, which further facilitates a reranker for performance improvement. Experimental results on the Gigaword dataset demonstrate our state-of-the-art performance for length-control summarization.

## introduction summarization systems aim to condense lengthy text into a shorter form, while preserving key information @xcite @xcite . recent studies underscore the importance of lengthcontrol summarization @xcite @xcite . for example, the google search engine limits webpage title displays to 63 characters, tweets have a 280-character cap, and scientific paper abstracts are typically restricted to a few hundred words. most previous text summarization methods rely on autoregressive (ar) models @xcite , which generate a summary word by word. for length control, ar models typically treat the summary length as an additional signal during both training and inference @xcite . however, such methods do not strictly control the generation length, and thus fall into the soft-constraint category. recently, non-autoregressive (nar) models @xcite have been applied to text summarization @xcite . unlike its autoregressive counterpart, an nar model predicts target words independently so that the length-control text generation problem can be decomposed into shared sub-problems, facilitating dynamic programming for efficient computation @xcite . such methods fall into the hard-constraint category. specifically, @xcite apply the connectionist temporal classification (ctc) model @xcite , which allows placeholders and repetitions during generation but removes them in a post hoc manner. however, these ctc-based methods tend to produce word-level extractive summaries due to their restricted modeling capacity. as a result, the generated summaries contain very few new words and mostly preserve the original word order @xcite . in this work, our first insight is to apply the directed acyclic transformer (dat, @xcite for length-control summarization. dat, originally introduced for machine translation, is a powerful nar approach that expands its canvas to allow multiple plausible text fragments and predicts links to connect them (the links along with the predicted words are called a path). in this way, dat is more flexible in predicting words and determining their order, resulting in better generation quality than ctc @xcite . for length-control decoding based on dat, we propose a sequence maximum a posteriori (seqmap) decoding objective, which marginalizes possible linked steps and seeks the most probable sequence satisfying the length budget. our seqmap is different from the existing map decoding objective @xcite that seeks the most probable path (links and words), which we refer to as pathmap. since dat training considers the marginalization of all possible links for a groundtruth sequence, our seqmap aligns with the training objective better (as it also performs marginalization), and is expected to surpass pathmap in length-control summarization. compared with pathmap, our seqmap is a more challenging decoding objective. the traditional pathmap performs arg max for both links and words, which can be reorganized according to prediction time steps and accomplished by dynamic programming in one pass. on the contrary, our seqmap performs arg max for word selection after summing over all possible linked steps, but max and sum operations cannot be directly swapped to decompose the overall problem by time steps, breaking down the dynamic programming algorithm. to this end, we propose an approximate algorithm to address seqmap, where we nevertheless decompose the objective by time steps. for each step, we make a greedy selection for the max operation, where we only consider a beam of highprobability sequences for the sum operation. further, we may apply a reranker by a pretrained bert model @xcite to select the best sequence in the beam for performance improvement. we perform extensive experiments using the gigaword @xcite and duc2004 (bommasani and cardie, 2020) datasets, following previous work in non-autoregressive length-control summarization @xcite . results show that both seqmap and pathmap outperform existing models based on ctc, which justifies our choice of the dat model. further, our approximate algorithm for seqmap outperforms pathmap even without the reranker, suggesting the superiority of the seqmap objective. our reranker further improves the summarization quality consistently, demonstrating its effectiveness. ## methodology in this section, we first present the directed acyclic transformer (dat) and the existing pathmap decoding method (subsection 2.1). then, we intro-duce our seqmap decoding objective, and develop a beam search-based dynamic programming algorithm to approach it (subsection 2.2). finally, we design a reranking process to select the best beam candidate (subsection 2.3). ## related work non-autoregressive (nar) models predict words independently, and are initially developed to increase the inference speed of neural machine translation @xcite @xcite @xcite . recently, nar models have been adapted for length-control summarization in our previous work @xcite , where we find that nar's independent word predictions allow the length-control tasks to be divided into several independent sub-tasks, resulting in an efficient exploration of the nar output space. therefore, we have developed dynamic programming algorithms based on the connectionist temporal classification (ctc) model @xcite . our work introduces a novel decoding algorithm that leverages the directed acyclic transformer (dat, @xcite . unlike ctc, which preserves the order of the source sequence @xcite , dat offers greater flexibility in word selection and generation order. while recognizing the value of existing datbased decoding methods @xcite for managing length, we identify their limitations and propose a new seqmap approach. our proposed reranker is inspired by the reranking methods in machine translation @xcite and summarization @xcite , where a list of n-best sequences are presented to an external model for scoring. ## conclusion this work proposes a novel seqmap decoding method based on the directed acyclic transformer 11579 (dat) for length-control summarization. our seqmap is a beam search-based dynamic programming algorithm, which bridges the gap between the training and inference of dat. experimental results on the gigaword and duc2004 datasets demonstrate the effectiveness of our seqmap approach.