Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarization (2022)

Abstract: Sentence summarization shortens given texts while maintaining core contents of the texts. Unsupervised approaches have been studied to summarize texts without ground-truth summaries. However, recent unsupervised models are extractive, which remove words from texts and thus they are less flexible than abstractive summarization. In this work, we devise an abstractive model based on reinforcement learning without ground-truth summaries. We formulate the unsupervised summarization based on the Markov decision process with rewards representing the summary quality. To further enhance the summary quality, we develop a multi-summary learning mechanism that generates multiple summaries with varying lengths for a given text, while making the summaries mutually enhance each other. Experimental results show that the proposed model substantially outperforms both abstractive and extractive models, yet frequently generating new words not contained in input texts.

## introduction the goal of sentence summarization is to enhance the readability of texts by reducing their lengths through word dropping, replacement, or paraphrasing. the applications of the task include subtitle generation @xcite and email summarization @xcite . an issue is that it is costly to have human editors write summaries for each text. hence, it is critical to develop an unsupervised model that does not require any human-written summaries. early models focus on abstractive summarization that generates words from a vocabulary set rather than extractive summarization, which merely selects words from texts. specifically, abstractive models have adopted autoencoder networks to summarize texts in an unsupervised manner (wang and * work done during internship at microsoft research asia. † corresponding author @xcite @xcite . in contrast, extractive models summarize texts by finding word combinations from texts, aiming at maximizing predefined scores (e.g., fluency of summaries) @xcite . despite their limited functionality, i.e., word selection, recent extractive models outperformed the abstractive models @xcite . despite the success of the extractive models, we argue that they have an inherent downside. the extractive models only select words from texts, and thus they cannot generate new words that can be effective for sentence summarization. for example, extractive models are unable to generate acronyms (e.g., pm) for words (e.g., prime minister) if the acronyms do not appear in texts. in contrast, abstractive models can resolve the limitation of extractive models. however, the summary quality of existing abstractive models is sometimes worse than a simple baseline, which simply truncates input texts from the beginning @xcite . this implies that existing abstractive models fall short of reducing text lengths while maintaining the summary quality. the aforesaid limitations of existing models motivate us to devise an abstractive model that produces high-quality summaries with generating new words not contained in input texts. this work employs reinforcement learning (rl) for unsupervised abstractive summarization 1 . rl enables a model to learn to summarize using rewards even though they are non-differentiable. our model generates high-quality summaries with considering 1) the semantic similarity between the generated summary and its corresponding input text, and 2) fluency of the generated summaries. notably, the semantic similarity is more robust to preserve core contents of input texts than the wordlevel reconstruction objective @xcite , which is adopted by existing abstractive models. moreover, we argue that the difficulty of summarization depends on the summary lengths (e.g., the shorter the summary, the more difficult it is to summarize). in this respect, we develop a multisummary learning mechanism that generates multiple summaries with varying lengths for a given text, while making the summaries mutually enhance each other. the main idea is to use a highquality summary of a certain length, which is easy to generate, to enhance the quality of a low-quality summary of another length, which is difficult to generate, rather than independently generating summaries in each length. specifically, we design the mechanism to make low-quality summaries semantically similar to high-quality ones. we also devise a pretraining task to obtain wellinitialized model parameters for the rl training. we first augment input texts by applying word-level perturbations and inserting length prompts, which indicate the lengths of the original texts. then, we train the model to reconstruct the original text from the augmented one, which makes the model learn to summarize and control the output length. by pretraining the model in this manner, our goal is to equip the model with essential abilities for summarization, which results in an improved summary quality after the rl training with the pretrained model. we dub our model multi-summary based reinforcement learning with pretraining (msrp). experiments show that msrp outperforms the abstractive and extractive baseline models in both automatic and human evaluations. we also analyze summaries generated by msrp to illuminate its benefits compared to the recent extractive models. ## conclusion this work employs the rl for unsupervised abstractive sentence summarization with the rewards representing the summary quality and length. we invent the multi-summary learning mechanism to make the summaries with varying lengths mutually enhance each other. in addition, we design the prompt-based text reconstruction task to further improve the rl training. experimental results show that msrp achieves the state-of-the-art summary quality on both automatic and human evaluation.