Advancing Precise Outline-Conditioned Text Generation with Task Duality and Explicit Outline Control (2024)

Abstract: Existing works on outline-conditioned text generation typically aim to generate text using provided outlines as rough sketches, such as keywords and phrases. However, these approaches make it challenging to control the quality of text generation and assess consistency between outlines and generated texts due to lack of clarity and rationality of the rough outlines. In this paper, we introduce a novel text generation task called , which requires generating stories based on specific , sentence-level outlines. To facilitate research on this task, we construct two new datasets, WPOG and CDM . We provide strong baselines based on fine-tuning models such as BART and GPT-2, and evaluating zero-shot performance of models such as ChatGPT and Vicuna. Furthermore, we identify an issue of imbalanced utilization of the outline information in the precise outline-conditioned generation, which is ubiquitously observed across fine-tuned models and zero-shot inference models. To address this issue, we propose an explicit outline utilization control approach and a novel framework that leverages the task duality between summarization and generation . Experimental results show that the proposed approaches effectively alleviate the issue of imbalanced outline utilization and enhance the quality of precise outline-conditioned text generation for both fine-tuning and zero-shot settings.

## introduction outline-conditioned text generation is a challenging but important task, which provides structured conditions (outlines) that aim to enhance content quality, customization, and generation efficiency for a variety of downstream text generation applications, e.g., story-telling and long-form questionanswering. it requires the model to not only generate fluent and coherent text but also ensure that the generated text aligns with the content, struc-ture, and properties (e.g., style) specified in the outline. existing works @xcite @xcite mainly regard a set of keywords or phrases as the "rough outline". the rough outline is usually extracted via heuristic methods @xcite and is overly abstractive, making it difficult to guarantee either quality or rationality of the rough outline, which in turn hampers quality of the generated text conditioned on the outline. moreover, the ambiguity and poor readability of the rough outline also complicate the evaluation of consistency between the outlines and generated text (e.g., simply asking the evaluator "which text is better at utilizing the keywords?" as in @xcite could be quite challenging to answer). to address these limitations of text generation conditioned on rough outlines, we introduce a novel task called precise outline-conditioned generation. as shown in figure 1 , this task takes the specific, sentence-level outline rather than keywords and phrases, which could improve the readability, clarity, and rationality of outlines and hence the quality of generated text and could better facilitate assessing consistency between the outline and generated text. compared to text generation conditioned on rough outlines, our proposed task poses greater challenges in three aspects: a) controllability: precise outlines inherently encode properties (e.g., style and attributes) that vary drastically with scenarios (e.g., stories, news reports). models need to grasp the properties and treat them as pre-given constraints to control text generation. b) faithfulness: the generated text needs to be faithful to the outline in terms of not only content but also relations among content in the outline. c) structure: the generated text needs to maintain a reasonable structure so that the key points in the generated text are well organized. to facilitate research on this new task, we provide two datasets wpog and cdm from storytelling and news report scenarios, respectively. we study how competitive models address these challenges by fine-tuning widely used pre-trained language models (lms) such as bart @xcite and gpt-2 @xcite for inference and also zero-shot inference with topperforming large language models (llms) such as chatgpt @xcite and vicuna @xcite foot_0 . the two datasets exhibit two drastically different styles, so we could evaluate how well models understand the inherent style and control generation conditioned on the style, i.e., handling the controllability challenge. we observe a common issue of imbalanced utilization of the outline across fine-tuning and zero-shot settings, which affects the structure of the generated text. specifically, the outline information is mostly used at the beginning of text generation and does not effectively guide the rest of the generation process. this issue indicates that the models struggle with the challenges of maintaining faithfulness and structure. we hypothesize that the imbalanced outline utilization issue may be caused by a combination of exposure bias due to teacher forcing (the model relies too heavily on previously predicted words) and decoding strategies (the model assigns the highest probabilities to the correct outline information), leading to early depletion of the outline. in order to accurately assess the faithfulness and structure quality of the generated text, we propose three metrics: distribution variation, peak-value distance, and consistency degree. we propose two general approaches to mitigate this imbalance issue. the first method explicitly controls the usage of the given outline throughout the generation process. the second method exploits the task duality between precise outlineconditioned generation and summarization and implements a dual learning strategy. experimental results on the two datasets with automatic and human evaluations show that both approaches alleviate the imbalance issue and improve generation quality. our contributions can be summarized as follows: (1) we introduce a novel precise outlineconditioned generation task. (2) we provide two datasets wpog and cdm to support research on this task. (3) we identify the imbalanced outline utilization issue and propose three evaluation metrics to gauge outline utilization. (4) to alleviate this imbalanced outline utilization issue, we propose two general approaches, explicit outline utilization control and unified dual-task learning. (5) experimental results demonstrate that our approaches effectively mitigate the imbalanced outline utilization issue and improve the generation quality for both fine-tuning and zero-shot settings. outline-conditioned generation there are a variety of works that take outline-conditioned generation as the critical step to long text generation. xu et al. (2018); yao et al. (2019); shao et al. (2019) adopt the bi-lstm as the backbone with reinforcement learning and variational inference techniques to optimize with keywords/phrases skeleton, respectively. @xcite modeling the semantic role labels (srl) as outlines by the self-attention mechanism and pointer mechanism @xcite . @xcite al. (2020); chen et al. (2021); wang et al. (2022) use pretrained language models to tackle with outlines in the form of keywords, eventgraph, and latent variables, respectively. @xcite propose a method to learn outline generation by reconstructing the summary, followed by generating segment-level text and concatenating them to obtain the full text. for zero-shot inference with llms, most existing works focus on improving the interaction or fluency with self-generated outlines. yang et al. (2022); zhou et al. (2023) apply the recursive prompting and revision to enhance gpt-3 and chatgpt over long-form storytelling, respectively. @xcite make a detailed outliner and a controller based on fudge @xcite to improve long-range plot coherence. in this paper, we consider both fine-tuning and zero-shot settings and propose unified strategies that could work for both scenarios. ## dataset creation to facilitate the study on the precise outlineconditioned generation task, we introduce two datasets, namely, writingprompt for outline-conditioned generation (wpog) and cnn/dailymail (cdm) @xcite , from two distinct storytelling and news report scenarios, respectively. wpog is constructed based on the english writ-ingprompt (wp) dataset @xcite , which includes human-written stories paired with writing prompts from an online forum. since the original wp does not provide outlines, we randomly select 5% of the samples and use gpt-4 3 to generate the 3 https://openai.com/research/gpt-4 outlines for original stories. moreover, we conduct the human evaluation to verify the reliability of generated outlines (the details and results are shown in appendix d). the wpog dataset is then built by pairing the generated outlines and the original writing prompts as input and the human-written stories as target references. to ensure fair comparisons between fine-tuning and zero-shot inference settings, we consider the limitations on the number of tokens output by bart and gpt-2 and exclude samples longer than 1024 tokens. we partition wpog into 80% for training, 10% for validation, and 10% for testing. cdm is an english dataset containing unique news stories (written by journalists with cnn and the daily mail) and the corresponding human-written highlights. for our task, the highlights are taken as outlines to represent the key plots of the news reports. we keep the original partition of training and validation sets as @xcite , while sampling 10% from the original test set as our test set for reasonable turnaround time for zeroshot evaluations of llms. we follow the previous work @xcite to obtain prompts and combine the prompt and outlines as input. ## experiments the details of dataset preprocessing, dataset statistics, experimental setup, and implementations are presented in appendix b. ## conclusion in this paper, we introduce a novel text generation task called precise outline-conditioned generation, which generates stories based on specific, sentencelevel outlines. we construct two new benchmark datasets, wpog and cdm, for this task. we provide strong baselines based on fine-tuning models such as bart and gpt-2, and evaluating zeroshot performance of models such as chatgpt and vicuna. we identify an issue related to the imbalanced utilization of the outline information in the precise outline-conditioned generation, which affects both fine-tuning and zero-shot models. to address this issue, we have proposed a unified explicit outline utilization control approach and a novel framework that leverages the task duality between summarization and generation. we have shown that our proposed approaches effectively alleviate the issue of imbalanced outline utilization and enhance the quality of precise outline-conditioned text generation for both fine-tuning and zero-shot settings. we hope that our work will inspire more research on this challenging and interesting task.