Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization (2022)

Abstract: Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation.

## introduction text summarization is an important natural language processing (nlp) task, aiming at generating concise summaries for given texts while preserving the key information. it has extensive real-world applications such as headline generation @xcite . in this paper, we focus on the setting of sentence summarization @xcite . state-of-the-art text summarization models are typically trained in a supervised way with large training corpora, comprising pairs of long texts and their summaries @xcite @xcite . however, such parallel data are expensive to obtain, preventing the applications to less popular domains and less spoken languages. unsupervised text generation has been attracting increasing interest, because it does not require parallel data for training. one widely used approach is to compress a long text into a short one, and to reconstruct it to the long text by a cycle consistency loss @xcite @xcite . due to the indifferentiability of the compressed sentence space, such an approach requires reinforcement learning (or its variants), which makes the training difficult @xcite . recently, @xcite propose an edit-based approach for unsupervised summarization. their model maximizes a heuristically defined scoring function that evaluates the quality (fluency and semantics) of the generated summary, achieving higher performance than cycle-consistency methods. however, the search approach is slow in inference because hundreds of search steps are needed for each data sample. moreover, their approach can only select words from the input sentence with the word order preserved. thus, it is restricted and may generate noisy summaries due to the local optimality of search algorithms. to address the above drawbacks, we propose a non-autoregressive approach to unsupervised summarization (naus). the idea is to perform search as in @xcite and, inspired by @xcite , to train a machine learning model to smooth out such noise and to speed up the inference process. different from @xcite , we propose to utilize non-autoregressive decoders, which generate all output tokens in parallel due to our following observations: we conducted experiments on gigaword headline generation @xcite and duc2004 @xcite datasets. experiments show that our naus achieves state-of-the-art performance on unsupervised summarization; especially, it outperforms its teacher (i.e., the search approach), confirming that naus can indeed smooth out the search noise. regarding inference efficiency, our naus with truncating is 1000 times more efficient than the search approach; even with dynamic programming for length control, naus is still 100 times more efficient than search and several times more efficient than autoregressive models. our naus is also able to perform length-transfer summary generation, i.e., generating summaries of different lengths from training. ## approach in our approach, we first follow @xcite and obtain a summary by discrete search towards a heuristically defined objective function ( §2.1). then, we propose a non-autoregressive model for the summarization task ( §2.2). we present the training strategy and the proposed length-control algorithm in §2.3. ## conclusion in this work, we propose a non-autoregressive unsupervised summarization model (naus), where we further propose a length-control decoding algorithm based on dynamic programming. experiments show that naus not only archives stateof-the-art unsupervised performance on gigaword headline generation and duc2004 datasets, but also is much more efficient than search methods and autoregressive models. appendices present additional analyses and length-transfer experiments. limitation and future work. our paper focuses on unsupervised summarization due to the importance of low-data applications. one limitation is that we have not obtained rigorous empirical results for supervised summarization, where the developed model may also work. this is because previous supervised summarization studies lack explicit categorization of summary lengths @xcite , making comparisons unfair and problematic @xcite . such an observation is also evidenced by @xcite , where the same model may differ by a few rouge points when generating summaries of different lengths. nevertheless, we have compared with @xcite in our setting and show the superiority of the naus under fair comparison. we plan to explore supervised summarization in future work after we establish a rigorous experimental setup, which is beyond the scope of this paper. other words, β s,t is the maximum probability of s tokens that are reduced to t words. in (13), β 1,0 refers to the probability of one token that is reduced to zero words, in which case the first predicted token can only be the blank token , corresponding to eqn. (9) with @xmath0. likewise, β 1,1 is the maximum probability of one token that is reduced to one word. thus, it is the probability of the most probable non-token, corresponding to eqn. (11) with @xmath1. eqn. (15) asserts that fewer tokens cannot be reduced to more words; it is used for mathematical derivations, but need not to be explicitly implemented in our algorithm in §2.3. the recursion variable β s,t is computed by in other words, the variable β s,t can inherit β s-1,t with a predicted blank token , corresponding to eqn. (9); or it can inherit β s-1,t-1 with a predicted non-token, corresponding to eqn. (11). specially, if @xmath2, then the second term has β s-1,-1 undefined, and thus is ignored in the max operation. we need the max operator to take the higher probability in the two cases, since β s,t is the maximum probability of s tokens being reduced to t words. this corresponds to eqn. (12) with beam size @xmath3. to sum up, our inductive calculation guarantees that β s,t is the exact maximum probability of max b:|b|=s,|γ (b)|=t p (b|x) for the desired length t with s generation slots; our algorithm (if not merging repeating tokens) gives the corresponding b s,t as argmax p (b|x) under the same constraints, concluding the proof of part (1). we consider generating a sentence of two words word p (w 1 |x) p (w 2 |x) i 0.39 0.1 like 0.4 0.9 coding 0.1 0 0.11 0 from the two prediction slots, i.e., @xmath4. apparently, the optimal sequence is "i like" with probability 0.39 • 0.9 = 0.351. however, the algorithm would predict b 1,1 = {"like"} because "like" is the most probably token in the first slot. then, our algorithm will give b 2,2 = {"like i"}, because it has to select a non-repeating token based on γ, yielding a non-optimal solution. it is noted that, if we do not merge repeating tokens as in γ , our algorithm will give the exact optimum "like like" in the above example. this shows that merging consecutive repeated tokens requires the decoding algorithm to correct early predictions, and thus, our dynamic programming becomes an approximate inference. nevertheless, our algorithm is able to generate a sequence of the desired length properly; its approximation happens only when the algorithm compares more repetitions with fewer s versus more s with fewer repetitions. such approximation is further alleviated by beam search in our dynamic programming. therefore, the proposed length-control algorithm is better than truncating a longer sentence; especially, our approach generates more fluent and complete sentences.