PLANET : Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation (2022)

Abstract: Despite recent progress of pre-trained language models on generating fluent text, existing methods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow. In this work, we propose PLANET, a novel generation framework leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. To guide the generation of output sentences, our framework enriches the Transformer decoder with latent representations to maintain sentence-level semantic plans grounded by bag-of-words. Moreover, we introduce a new coherence-based contrastive learning objective to further improve the coherence of output. Extensive experiments are conducted on two challenging long-form text generation tasks including counterargument generation and opinion article generation. Both automatic and human evaluations show that our method significantly outperforms strong baselines and generates more coherent texts with richer contents.

## introduction neural sequence-to-sequence (seq2seq) models are dominant methods for text generation nowadays, which are trained to maximize the log-likelihood over targets in an end-to-end fashion @xcite . recently, pre-trained methods such as gpt-2 @xcite and bart @xcite have achieved promising results by leveraging large-scale data. while these models can generate fluent results, they still fall short of producing coherent long-form texts with multiple sentences @xcite . long text generation, especially opinion generation, usually requires the model to ( 1 ) conduct proper content selection and ordering (i.e., "what to say and when to say it") to form a coherent highlevel logical flow, and (2) appropriately reflect the ## conclusion we present a novel generation framework to dynamically conduct content planning and surface realization in large autoregressive transformers by leveraging self-attention and high-level latent representations. the latent representations are grounded by bag-of-words that measures the overall semantic plan of each target sentence. we further introduce a novel coherence-based contrastive objective with different negative sample construction strategies to improve output coherence. experiment results on two opinion text generation tasks demonstrate that our model can generate high-quality outputs with better coherence and content richness. cheung. 2019. a cross-domain transferable neural coherence model. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 678-687, florence, italy. association for computational linguistics. lili yao, nanyun peng, ralph weischedel, kevin knight, dongyan zhao, and rui yan. 2019. planand-write: towards better automatic storytelling. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 7378-7385.