Sentence-Permuted Paragraph Generation (2021)

Abstract: Generating paragraphs of diverse contents is important in many applications. Existing generation models produce similar contents from homogenized contexts due to the fixed left-to-right sentence order. Our idea is permuting the sentence orders to improve the content diversity of multi-sentence paragraph. We propose a novel framework PermGen whose objective is to maximize the expected log-likelihood of output paragraph distributions with respect to all possible sentence orders. PermGen uses hierarchical positional embedding and designs new procedures for training, and decoding in the sentence-permuted generation. Experiments on three paragraph generation benchmarks demonstrate PermGen generates more diverse outputs with a higher quality than existing models.

## introduction paragraph generation is an important yet challenging task. it requires a model to generate informative and coherent long text that consists of multiple sentences from free-format sources such as a topic statement or some keywords @xcite . typical paragraph generation tasks include story generation @xcite , news generation @xcite , scientific paper generation @xcite , etc. recent advances in natural language generation models such as transformer @xcite and bart @xcite ) have demonstrated attractive performance of generating text paragraphs. an important desired property of modelgenerated paragraphs is diversity -given the same source, an intelligent model is expected to create a variety of paragraphs in terms of content, semantic style, and word variability @xcite . for example, a story generation § our code and output files are available at https:// github.com/wyu97/permgen . ## related work paragraph generation. the source can be either structured or unstructured such as database records @xcite , knowledge graphs @xcite , images @xcite , and keywords @xcite . the expected outputs typically are stories @xcite , essays @xcite , news articles @xcite , or scientific papers @xcite . this task poses unique challenges as it aims at generating coherent and diverse long-form texts. our framework can use various forms of input such as a story title, keywords, and keyphrases, which can be generalized to broad domains. ## experiments we conduct experiments on three text generation tasks: story generation, news generation, and paper abstract generation. for all tasks, we compare permgen with multiple baseline models on diversity and accuracy of their generated texts. we also perform human evaluation on story generation. ## conclusions in this paper, we proposed a novel sentencepermuted paragraph generation model, permgen. permgen maximizes the expected log likelihood of output paragraph w.r.t. all possible sentence orders. experiments on three paragraph generation tasks demonstrated that permgen outperformed original transformer by generating more accurate and diverse text. the result is consistent on various transformer models and decoding methods.