Learning Implicit Text Generation via Feature Matching (2020)

Abstract: Generative feature matching network (GFMN) is an approach for training state-of-the-art implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer.

## introduction generative feature matching networks (gfmns) @xcite has been recently proposed for learning implicit generative models by performing moment matching on features from pre-trained neural networks. this approach demonstrated that gfmn could produce state-of-the-art image generators while avoiding instabilities associated with adversarial learning. similarly to training generative adversarial networks (gans) @xcite , gfmn training requires to backpropagate through the generated data to update the model parameters. this backpropagation through the generated data, combined with adversarial learning instabilities, has proven to be a compelling challenge when applying gans for discrete data such as text. however, it remains unknown if this is also an issue for feature matching networks since the effectiveness of gfmn for sequential discrete data has not yet been studied. in this work, we investigate the effectiveness of gfmn for different text generation tasks. as a first contribution, we propose a new formulation of gfmn for unconditional sequence generation, which we name sequence-gfmn or seqgfmn for short, by performing token level feature matching. seqgfmn has a stable training because it does not concurrently train a discriminator, which in principle could easily learn to distinguish between one-hot and soft one-hot representations. as a result, we can use soft one-hot representations that the generator outputs during training without using the gumbel softmax or reinforce algorithm as needed in gans for text. additionally, different from gans @xcite , seqgfmn can produce meaningful text without the need of pre-training the generator with maximum likelihood estimation (mle). we perform experiments using bidirectional encoder representations from transformers (bert), glove, and fasttext as our feature extractor networks. we use two different corpora, and assess both the quality and diversity of the generated texts with three different quantitative metrics: bleu, self-bleu and fréchet infersent distance @xcite . additionally, we show that the latent space induced by seqgfmn contains semantic and syntactic structure, as evidenced by interpolations in the z space. our second contribution consists in proposing a new strategy for class-conditional generation with gfmn. the key idea here is to perform class-wise feature matching. we apply seqgfmn to perform sentiment-based conditional generation using the yelp reviews dataset, and assess its performance using classification accuracy, bleu, and self-bleu. finally, as a third contribution, we demonstrate that the feature matching loss is an effective approach to perform distribution matching in the context of unsupervised text style transfer (utst). most previous work on utst adapts the autoencoder framework by adding an additional loss term: adversarial loss or back-translation loss. our method consists in replacing the adversarial and back-translation loss with style-wise feature matching. our experimental results indicate that the feature matching loss produces better results than the traditionally used losses. and σ j,t p data ,ℓ on the entire training data. during training, we generate a minibatch of fake data by passing the gaussian noise vector through the generator. the fixed feature extractor e is used to extract features on the output of the generator at a per-token level. the loss is then computed, as mentioned in eq. 1. the parameters θ of the generator g are optimized using stochastic gradient descent. note that the network e is used for feature extraction only and is kept fixed during the training of g. similar to (dos santos et al., 2019), we use adam moving average, which allows us to use small minibatch sizes. fig. 1 illustrates se-qgfmn training; note that we use mean matching only for brevity, in practice we match both mean and diagonal covariance. in our seqgfmn framework, the output of the generator g is a sequence x of soft one-hot representations, { w1 , w2 , ..., wt }, where each element wi consists in the output of the softmax function at token i. in the feature extractor e, these soft onehot representations are multiplied by an embedding matrix to generate soft embeddings, which are then fed to the following layers of e. ## experiments and results datasets: we evaluate our proposed approach on three different english datasets: mscoco @xcite , emnlp 2017 wmt news dataset @xcite , and yelp reviews dataset @xcite . both coco and wmt news datasets are used for unconditional models, while yelp reviews is employed to evaluate classconditional generation and unsupervised text style transfer. feature extractors for textual data: we experiment with different feature extractors that generate token-level representations. we use word embeddings from glove @xcite and fasttext @xcite as representatives of shallow (cheap-to-train) architectures. as a representative of large, deep feature extractor we use bert @xcite . @xcite demonstrated that the features extracted by bert can boost the performance of diverse nlp tasks. our hypothesis is that bert features are informative enough to allow the training of (cross-domain) text generators with the help of feature matching. metrics: in order to evaluate the diversity and quality of texts of the unconditional generators we use three metrics bleu @xcite , self-bleu @xcite and fréchet infersent distance, fid @xcite . additionally, for class-conditional generation and unsupervised text style transfer, we report accuracy scores from a cnn sentiment classifier trained on the yelp. ## conclusion we presented new implicit generative models based on feature matching loss that are suitable for unconditional and conditional text generation. our results demonstrated that backpropagating through discrete data is not an issue for the training via matching distributions at the token level. seqgfmn can be trained from scratch without the need for rl or gumbel softmax. this approach has allowed us to create effective models for unconditional generation, class-conditional generation, and unsupervised text style transfer. we believe this work opens a new competitive avenue in the area of implicit generative models for sequential data.