Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task (2020)

Abstract: Despite the recent success of contextualized language models on various NLP tasks, language model itself cannot capture textual coherence of a long, multi-sentence document (e.g., a paragraph). Humans often make structural decisions on what and how to say about before making utterances. Guiding surface realization with such high-level decisions and structuring text in a coherent way is essentially called a planning process. Where can the model learn such high-level coherence? A paragraph itself contains various forms of inductive coherence signals called self-supervision in this work, such as sentence orders, topical keywords, rhetorical structures, and so on. Motivated by that, this work proposes a new paragraph completion task PARCOM; predicting masked sentences in a paragraph. However, the task suffers from predicting and selecting appropriate topical content with respect to the given context. To address that, we propose a self-supervised text planner SSPlanner that predicts what to say first (content prediction), then guides the pretrained language model (surface realization) using the predicted content. SSPlanner outperforms the baseline generation models on the paragraph completion task in both automatic and human evaluation. We also find that a combination of noun and verb types of keywords is the most effective for content selection. As more number of content keywords are provided, overall generation quality also increases.

## introduction one may think textual coherence can be achieved from a gigantic language model trained on massive data. this might be true in simple cases, such as generating short replies @xcite , but not in a long, multi-sentence generation. this is mainly because per-word predictions from the autoregressive models can not capture the long-term flow of text, while humans often make structural decisions on what and how to say about before they speak @xcite @xcite @xcite . guiding the surfacelevel realization with such high-level decisions and coherently structuring output text is called a planning process. where can the model learn such high-level decisions related to long-term coherence? a written paragraph itself can be a pot of golden resources, containing various forms of inductive coherence signals. different types of coherence signals in a paragraph have been studied and used in many different ways: a sequence of words or sentences @xcite , a discourse structure of a text @xcite @xcite , an order of sentences @xcite , topic introduction, co-reference, a sequence of events @xcite , and more. in this work, we primarily focus on the effect of topical content in text planning. despite the recent advances of contextualized language models @xcite , the lack of appropriate tasks makes it difficult to evaluate generation models' long-term coherence. prior tasks fall into classification or ranking problems, such as narrative close task @xcite , sentence ordering @xcite , and next sentence prediction @xcite . some recent works focused on designing generation tasks: story generation @xcite ), text infilling @xcite @xcite , or paragraph bridging @xcite . however, most of them suffer from predicting appropriate topical content given limited context, due to the limited usage of self-supervision signals from the paragraph. this work proposes a new open-ended paragraph completion task; parcom; predicting the masked sentences in a paragraph. unlike the prior works, our task uses two effective ways of self-supervision learnt from a written paragraph itself: (1) we augment more training instances via permutation masking and (2) resolve the context sparsity problem by providing a set of ground-truth content keywords and predicting them directly from context at testing time. for the task, we propose a self-supervised text planner (ssplanner) that explicitly predicts content keywords (content prediction) from context and guides the pretrained language model (surfacerealization) using the predicted content. the distribution of predicted keywords is then combined with the distribution of words in the language model using copy mechanism @xcite . the predicted content keywords are an approximation of topical intents by the generator, providing a hint to guide the surface realizer to bridge the coherency gap between the given context and text to generate. overall, ssplanner combines two advantages; micro-level language fluency from the pre-trained language model (bottom-up) and macro-level content choice controlled by the macro-level planning (top-down). our experiment shows that ssplanner achieves significant improvements over the baselines in both automatic and human evaluation. ## related work we first categorize a wide range of long-term coherent generation tasks ( c ≈ t: paraphrasing is transforming surface patterns of text while preserving its semantics. @xcite used variational autoencoders for surface realization with a latent bag of words model for differentiable content planning, where content to generate itself is given in context, not requiring any content planning. c ⊥ ⊥ t: story generation @xcite ), text infilling @xcite , paragraph bridging @xcite , and our proposed parcom are very challenging tasks where context and target have no overlap (open-ended), but they should be coherently connected. fedus et al. (2018); huang et al. (2019) developed various methods for text infilling task. very similar to our task, @xcite developed language models informed by discourse relations on the bridging task; given the first and last sentences, predicting the intermediate sentences (bidirectional flow). however, they did not explicitly predict content words given context nor use them as a self-supervision signal in training. unlike random masking in @xcite 2019); huang et al. (2019 , we propose a better data augmentation training method via permutation masking. ## parcom: paragraph completion our task is motivated by the recently proposed task; paragraph bridging @xcite , predicting intermediate sentences of a paragraph, given the first and the last sentences. to prevent generation becoming too divergent from the context in story or prompt generation @xcite , the bridging task restricts generation to end with the last sentence given, provided as an ending goal for generation. however, in the bridging task, the objective is to generate text by coherently linking the two extreme sentences, making the task itself too challenging even for human 1 . for instance, the first and last sentences are too sparse to generate multiple (from 2 to 5) target sentences, increasing divergence of generation exponentially. also, data usage in @xcite is very inefficient; training a single instance per paragraph. to address those issues, we propose a new paragraph completion task parcom by maximizing self-supervision presented in a paragraph itself (figure 1 ). we describe two types of selfsupervisions: (1) masking a fixed-length of consecutive sentences in any position over a paragraph to maximize usage of a paragraph and (2) extracting partial keywords of the masked text as plan keywords to resolve the content sparsity problem. mainly, we learn the patterns between the context and the plan keywords in training and at testing time predict the plan keywords, and guide the surface generator ( §4). ## self-supervised text planning (ssplanner) ssplanner has various self-supervision modules that learn coherence signals from a paragraph itself: surface realizer (language model) by learning from a sequence of words, next sentence predictor by learning from a sequence of two consecutive sentences, sentence position embeddings by learning from an order of context sentences, plan predictor by learning from the relationship between the given context and important keywords used in the generation of the target text, and content guidance by learning from whether the predicted plan keywords are used or not in the target (see figure 2 ). our planner is motivated by the two-stage generation framework @xcite @xcite . while in prior works, the content is explicitly given from the dataset or task itself, our plan predictor in ssplanner predicts the plan keywords only from the given context, by learning the topical relationship between context and content in target from training data. given l length of a paragraph s 1 ..s l where each sentence s consists of a n number of words @xmath0..w n , parcom splits it into the context sentences @xmath1..s j-1 , s j+t ..s n and t target sentences to predict @xmath2..s j+t-1 . for each target sentence, p number of plan keywords k j,1 ..k j,p for arbitrary target sentence s j are given only at training time. the plan keywords are chosen from the entire vocabulary v w and later combined with word distribution from the language model. we describe each selfsupervision module in ssplanner as follows: surface realization with pre-trained language models. we use two different types of transformer-based language models: bert @xcite and gpt2 @xcite . while gpt2 is trained on bidirectionally tied language modeling, bert is trained on masked language modeling. for bert, we use the sequential sampling method @xcite . using them, we encode context x and output the hidden representation h j,@xmath3,i , x @xmath4,i) ) for j th word in i th sentence, where @xmath5, gpt2} is the transformer language model. we then output the sentence vector h i by averaging all word vectors in a sentence. ## conclusion a written paragraph itself contains various inductive coherence signals to be learned through selfsupervision. motivated by this, we propose a paragraph completion task for measuring textual coherence from a long document using different types of self-supervision signals. to solve the task, we propose a text planner ssplanner that explicitly predicts topical content keywords, and then guides the surface generator using the predicted plan keywords. ssplanner consists of different kinds of self-supervision modules: sentence positions, a sequence of words or sentences, and the topical relationship between context and target. our selfsupervised planning, in addition to other types of planning (e.g., discourse, goals, coreference, tenses) can be an important step toward modeling a long-term coherence in text generation. our results suggest several promising directions: although our ablation tests show the effect of each self-supervision module, types of plan keywords, and the amount of keywords with respect to generation quality, there are more spaces to explore in self-supervised text planning. for example, one can study the generation quality with respect to the position of the target sentences (beginning, middle, end), the comparison of plan keywords predicted by human and system, the effect of data augmentation by their positions (e.g., masking the only middle), the generation quality with respect to the ratio between masked and unmasked sentences, and more. second, we can extend the set of plan keywords to be more structured like a discourse tree. for instance, one can write a simple structure like "(causality (elaborate (buy, coffee)) (pay, tip, 12 dollars))" then the system can generate a long, coherent text reflected by the structure. predicting such structural plans from context and imposing them into the generator would be a potential direction for future work. last, text planning is a cognitive function commonly used in human language generation. to generate more human-like utterances, different planning stages should be simultaneously combined together @xcite , such as abstractive planning, strategic planning, coherence planning, and diversity planning. combining the heterogeneous planning systems will be a crucial step towards developing a human-like language generation.