Improving Adversarial Text Generation by Modeling the Distant Future (2020)

Abstract: Auto-regressive text generation models usually focus on local fluency, and may cause inconsistent semantic meaning in long text generation. Further, automatically generating words with similar semantics is challenging, and hand-crafted linguistic rules are difficult to apply. We consider a text planning scheme and present a model-based imitation-learning approach to alleviate the aforementioned issues. Specifically, we propose a novel guider network to focus on the generative process over a longer horizon, which can assist next-word prediction and provide intermediate rewards for generator optimization. Extensive experiments demonstrate that the proposed method leads to improved performance.

## introduction text generation is an important area of investigation within machine learning. recent work has shown excellent performance on a number of tasks, by combining reinforcement learning (rl) and generative models. example applications include image captioning @xcite ), text summarization @xcite @xcite , and adversarial text generation @xcite @xcite @xcite . the sequence-to-sequence framework (seq2seq) @xcite is a popular technique for text generation. however, models from such a setup are typically trained to predict the next token given previous ground-truth tokens as input, causing what is termed exposure bias @xcite . by contrast, sequence-level training with rl provides an effective means of solving this challenge, by treating text generation as a sequential decision-making problem. by directly optimizing an evaluation score (cumulative rewards) @xcite , state-of-the-art results have been ob-tained in many text-generation tasks @xcite . however, one problem in such a framework is that rewards in rl training are particularly sparse, since a scalar reward is typically only available after an entire sequence has been generated. furthermore, the recurrent models focus more on local fluency, and may cause inconsistent semantic meanings for long text generation. for rl-based text generation, most existing works rely on a model-free framework, which has been criticized for its high variance and poor sample efficiency @xcite . on the other hand, while model-based rl methods do not suffer from these issues, they are usually difficult to train in complex environments. further, a learned policy is usually restricted by the capacity of an environment model. recent developments on model-based rl @xcite @xcite combine the advantages of these two approaches, and have achieved improved performance by learning a model-free policy, assisted by an environment model. in addition, model-based rl has been employed recently to solve problems with extremely sparse rewards, with curiosity-driven methods @xcite . in this paper, we propose a model-based imitation-learning method to overcome the aforementioned issues in text-generation tasks. our main idea is to employ an explicit guider network to model the generation environment in the feature space of sentence tokens, used to emit intermediate rewards by matching the predicted features from the guider network and features from generated sentences. the guider network is trained to encode global structural information of training sentences, and thus is useful to guide next-token prediction in the generative process. within the proposed framework, to assist the guider network, we also develop a new type of self-attention mechanism to provide high-level planning-ahead information and maintain consistent semantic meaning. our experimental results demonstrate the effectiveness of proposed methods. starting from the initial state s 0 , a recurrent neural network (rnn) produces a sequence of states (s 1 , . . . , s t ) given an input sentence-feature representation (e(y 1 ), . . . , e(y t )), where e(•) denotes a word embedding function mapping a token to its ddimensional feature representation. the states are recursively updated with a function known as the cell: s @xmath0, e(y t )). one typically assigns the following probability to an observation y at location t: p(y|@xmath1. together (g, h) specifies a probabilistic model π, i.e., log π(y ) = t log p(y t |@xmath2. to train the model π, one typically uses maximum likelihood estimation (mle), via minimizing the cross-entropy loss, i.e., j mle (π) = -e[log π(y )]. in order to generate sentence y s from a (trained) model, one iteratively applies the following operations: model-based imitation learning text generation can be considered as an rl problem with a large number of discrete actions, deterministic transitions, and deterministic terminal rewards. it can be formulated as a markov decision process (mdp) @xmath3, a, p, r, γ , where s is the state space, a is the action space, p is the deterministic environment dynamics, r(s, y) is a reward function, and γ ∈ (0, 1) is the discrete-time discount factor. the policy π φ , parameterized by φ, maps each state @xmath4. the objective is to maximize the expected reward: in model-based imitation learning @xcite , a model is built to make predictions for future state s t+ t conditioned on the current state foot_0 , which can be used for action selection, e.g., next-token generation. this model is typically a discrete-time system, taking the current state-action pair (s t , y t ) as input, and outputting an estimate of the future state s t+ t at time t + t. at each step t, y t is chosen based on the model, and the model will re-plan with the updated information from the dynamics. this control scheme is different from a standard model-based method, and is referred to as model-predictive control (mpc) @xcite . note that in our setting, the state in rl typically corresponds to the current generated sentences y 1,...,t instead of the rnn state of generator (decoder). ## proposed model the model is illustrated in figure 1 , with an autoeocoder (ae) structure for sentence feature extraction and generation. the encoder is shared for sentences from both training data and generated data, as explained in detail below. overall, text generation can be formulated as an imitationlearning problem. at each timestep t, the agent, also called a generator (which corresponds to the lstm decoder), takes the current lstm state as input, denoted as s t . the policy π φ (•|s t ) parameterized by φ is a conditional generator, to generate the next token (action) given s t , the observation representing the current generated sentence. the objective of text generation is to maximize the total reward as in (4). we detail the components for our proposed model in the following subsections. ## extension to non-parallel text style transfer as illustrated in figure 2 , our framework naturally provides a way for style transfer, where the guider network plays the role of style selection, and the generator only focuses on maintaining content without considering the styles. to make the guider network focus on the guidance of styles, we assign the label l as the initial state s g 0 of the guider network. specifically, at each step t, we feed the current sentence representation f t and label l into the guider network: for the generator, we put an adversarial regularizer on the encoded latent s 0 (x) and penalize it if it contains the sentiment information, by maximizing the entropy, i.e., max l p(l| s 0 (x)) log p(l| s 0 (x)), where p is a pre-trained classifier. intuitively, the generator gives candidate words represented by o t , while the guider makes a choice implicitly by w t based on the sentiment information. the sentiment information is contained in w t , while the content of the original sentence is represented by o t . to achieve styletransfer, one feeds the original sentence x with the target style label l to get the transferred sentence y with style l. following previous work @xcite @xcite , we adopt a classifier as the discriminator and the soft-argmax approach (kusner and miguel, 2016) for the update of generator instead of policy gradient @xcite . ## related work we first review related works that combine rl and gan for text generation. as one of the most rep-resentative models in this direction, seqgan @xcite adopts monte-carlo search to calculate rewards. however, such a method introduces high variance in policy optimization. there are a number of works proposed subsequently to improve the reward-generation process. for example, rankgan @xcite proposes to replace the reward from the gan discriminator with a rankingbased reward, @xcite modifies the gan objective and proposes techniques to reduce gradient variance, maskgan @xcite uses a filling technique to define a q-value reward for sentence completion, relgan @xcite uses a relational memory based generator for the long-distance dependency modeling, fm-gan @xcite uses a feature mover distance to match features of real and generated sentences inspired by optimal transport @xcite , and leakgan @xcite tries to address the sparse-reward issue for long-text generation with hierarchical rl by utilizing the leaked information from a gan discriminator. one problem of leakgan is that it tends to overfit the training data, yielding generated sentences that are often not diverse. by contrast, by relying on a model-based imitation learning approach, our method learns global-structure information, which generates more-diverse sentences, and can be extended to conditional text generation. zhang et al. (2020) designed a differentiable nested wasserstein distance for semantic matching, which can be applied for further improvement. rl techniques can also be used in other ways for text generation @xcite . note that all these methods can only obtain reward after a whole sentence is generated. planning techniques in rl have also been explored to improve text generation @xcite . @xcite introduced the selfimitation scheme to exploit historical high-quality sentences for enhanced exploration. compared to these related works, the proposed guider network can provide a planning mechanism and intermediate rewards.