Stylized Text Generation: Approaches and Applications (2020)

Abstract: Text generation has played an important role in various applications of natural language processing (NLP), and kn recent studies, researchers are paying increasing attention to modeling and manipulating the style of the generation text, which we call stylized text generation. In this tutorial, we will provide a comprehensive literature review in this direction. We start from the definition of style and different settings of stylized text generation, illustrated with various applications. Then, we present different settings of stylized generation, such as style-conditioned generation, style-transfer generation, and style-adversarial generation. In each setting, we delve deep into machine learning methods, including embedding learning techniques to represent style, adversarial learning, and reinforcement learning with cycle consistency to match content but to distinguish different styles. We also introduce current approaches to evaluating stylized text generation systems. We conclude our tutorial by presenting the challenges of stylized text generation and discussing future directions, such as small-data training, non-categorical style modeling, and a generalized scope of style transfer (e.g., controlling the syntax as a style).

## tutorial introduction text generation has played an important role in various applications of natural language processing (nlp), such as paraphrasing, summarization, and dialogue systems. with the development of modern deep learning techniques, text generation is usually accomplished by a neural decoder (e.g., a recurrent neural network or a transformer), which generates a word at a time conditioned on previous generated words. the decoder can be further conditioned on some source information, such as a source language sentence in machine translation, or a previous utterance in dialogue systems. in recent studies, researchers are paying increasing attention to modeling and manipulating the style of the generation text, which we call stylized text generation in this tutorial. the goal is to not only model the content of text (in traditional text generation), but also control some "style" of the text, for example, the persona of a speaker in a dialogue @xcite , or the sentiment of product reviews @xcite . stylized text generation is related to various machine learning techniques, for example, embedding learning techniques to represent style @xcite , adversarial learning and reinforcement learning with cycle consistency to match "content" but to distinguish different styles @xcite @xcite ; very recent work is even able to disentangle latent features in an unsupervised way @xcite . in this tutorial, we will provide a comprehensive literature review on stylized text generation. we start from the definition of style and different settings of stylized text generation, illustrated with various applications. in the second part, we will describe style-conditioned text generation. in this category, style serves as a certain type of source information, which the decoder is conditioned on. we describe three types of approaches: (1) embeddingbased techniques that capture the style information by real-valued vectors, which can be used to condition a language model @xcite or concatenated with the input to a decoder @xcite (2) approaches that encode both style and content in the latent space @xcite @xcite . we will discuss techniques that structure latent space to encode both style and content, and include gaussian mixture model variational autoencoders (gmm-vae) @xcite @xcite , conditional variational autoencoders (cvae) @xcite , and adversarially regularized autoencoders (arae) @xcite . (3) approaches with multiple style-specific decoders @xcite . we highlight several applications including personabased dialogue generation @xcite and creative writing @xcite @xcite . next, we will introduce evaluation methods for style-conditioned text generation. we will present the current practice in the literature, involving both human evaluation and automatic metrics. a few important evaluation aspects include the success of being in the target style, the preservation of content information, as well as language fluency in general. in the third part, we will focus on style-transfer text generation. given an input sentence of a certain style, the goal of style transfer is to synthesize a new sentence that has the same content but with different styles. particularly, style-transfer text generation can be categorized into three settings: next, we will discuss style adversarial text generation @xcite . the setting of adversarial attacks is similar to style transfer in that it aims to change the style classifier's prediction. however, the synthesized sentence in this setting should in fact keep the actual style as humans perceive, but "fool" the style classifier. thus, it is known as the adversarial attack. we will discuss style adversarial generation in the character level, the word level, as well as the sentence level. techniques include discrete word manipulation and continuous latent space manipulation. finally, we will conclude our tutorial by presenting the challenges of stylized text generation and discussing future directions, such as smalldata training, non-categorical style modeling, and a generalized scope of style transfer (e.g., controlling the syntax as a style @xcite ). by the end of the tutorial, the audience will have a systematic view of different settings of stylized text generation, understand common techniques to model and manipulate the style of text, and be able to apply existing approaches to new scenarios that require stylized text generation. our tuto-rial also investigates stylized generative models in non-nlp domains, and thus would inspire future nlp studies in this direction.