Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries (2022)

Abstract: The difficulty of generating coherent long texts lies in the fact that existing models overwhelmingly focus on the tasks of local word prediction, and cannot make high level plans on what to generate or capture the high-level discourse dependencies between chunks of texts. Inspired by how humans write, where a list of bullet points or a catalog is first outlined, and then each bullet point is expanded to form the whole article, we propose SOE , a pipelined system that involves of summarizing, outlining and elaborating for long text generation: the model first outlines the summaries for different segments of long texts, and then elaborates on each bullet point to generate the corresponding segment. To avoid the labor-intensive process of summary soliciting, we propose the reconstruction strategy, which extracts segment summaries in an unsupervised manner by selecting its most informative part to reconstruct the segment. The proposed generation system comes with the following merits: (1) the summary provides high-level guidance for text generation and avoids the local minimum of individual word predictions; (2) the high-level discourse dependencies are captured in the conditional dependencies between summaries and are preserved during the summary expansion process and (3) additionally, we are able to consider significantly more contexts by representing contexts as concise summaries. Extensive experiments demonstrate that SOE produces long texts with significantly better quality, along with faster convergence speed.

## introduction despite that recent large-scale pretrained language models (plms) @xcite @xcite are able to produce highquality passages that can be hardly recognized by humans @xcite , most of the generated "good" texts are within very limited length, e.g. hun-dreds of tokens for most cases @xcite , thus generating coherent long texts remains a challenge @xcite . the difficulty lies in the fact that existing models generate texts in a word-by-word manner: predicting each subsequent token given its preceding contexts. this word-by-word strategy overwhelmingly focuses on predicting local words, and cannot make high level plans on what to generate. the strategy results in the fact that long texts generated by current models are usually repetitive, generic and self-contradictory @xcite . to address these issues, the coarse-to-fine generation strategy is proposed @xcite @xcite . in coarse-to-fine generation, a list of keywords or a short prompt is first generated, serving as a summary of the original text. the prompt is then fed to a seq2seq model as an input to output the complete text. the coarse-to-fine generation strategy significantly improves generation over the word-byword strategy, but still suffers from the following shortcomings: (a) limited capacity of the prompt: a single keyword list or prompt does not have enough capacity to summarize all the text of long passages, since long texts are usually consists of several parts, each of which focuses on a specific aspect or topic @xcite . the usage of the coarse-to-fine generation strategy is thus limited to texts that can be summarized by a single prompt (e.g., short stories). this explains why text length generated by the progressive generation model is still limited, e.g., the introduced writing prompts dataset in @xcite has an average length of stories around 735, and the average length of prompts is 28; (b) ignorance of high-level discourse dependency: the coarse-to-fine generation strategy does not capture discourse-level dependencies @xcite , which handle the highlevel information flow and interactions between segments of texts. the ignorance of discourse-level dependencies results in texts lacking for coherence. humans write in a hierarchical top-down manner: before writing a thousand-word-long essay, a human usually first prepares a list of bullet points or a catalogue, and then expands them to form the whole article. the sentence-level coherence between these bullet points is preserved when the bullet points are expanded, providing guarantees that the full text is coherent. to mimic this top-down manner in human writing, we propose soe, a pipelined system that involves of summarizing, outlining and expanding for long text generation: the model first outlines the summaries for different segments of long texts, which actually mimics the process of humans outlining bullet points; next, the model elaborates on each bullet point to generate the corresponding segment. the proposed strategy comes with the following merits: (a) since each segment is associated with its own summary rather than the entire text sharing a single prompt, the capacity of summaries to reconstruct the full text can be guaranteed; (b) the conditional generation probability between summaries captures the high-level discourse dependencies, and these dependencies are preserved when they are expanded to segments. this naturally resolves the incapability of modeling discourse-level dependencies in the coarse-to-fine generation approach. (c) this model is able to consider significantly larger amount of contexts by representing chunks of contexts as concise summaries. empirically, we do not readily have summaries for segments in hand. the model thus needs to learn to summarize in an unsupervised manner. to this end, we propose the reconstruction strategy, which extracts segment summaries by selecting its most informative part to reconstruct the segment. extensive experiments demonstrate that soe produces long texts with significantly better quality than existing baselines. ## experiments in this section, we present experiment results. for different methods to generate summaries, we find that the performance of reconstruction consistently outperforms the rest in our preliminary results. we ## conclusion in this paper, we propose a two-step hierarchical generation strategy for long-text generation: the model first generates the summary for each segment conditioning on previous summaries, and next, each summary is expanded to form the full text segment. the proposed strategy provides highlevel guidance for local text generation, and enables high-level discourse dependencies to be captured. extensive experiments demonstrate that soe produces long texts with significantly better quality,