Making Use of Latent Space in Language GAN s for Generating Diverse Text without Pre-training (2021)

Abstract: Generating diverse texts is an important factor for unsupervised text generation. One approach is to produce the diversity of texts conditioned by the sampled latent code. Although several generative adversarial networks (GANs) have been proposed thus far, these models still suffer from mode-collapsing if the models are not pre-trained. In this paper, we propose a GAN model that aims to improve the approach to generating diverse texts conditioned by the latent space. The generator of our model uses Gumbel-Softmax distribution for the word sampling process. To ensure that the text is generated conditioned upon the sampled latent code, reconstruction loss is introduced in our objective function. The discriminator of our model iteratively inspects incomplete partial texts and learns to distinguish whether they are real or fake by using the standard GAN objective function. Experimental results using the COCO Image Captions dataset show that, although our model is not pre-trained, the performance of our model is quite competitive with the existing baseline models, which requires pre-training.

## introduction generative adversarial networks (gans) @xcite have recently received significant attention in the field of unsupervised text generation, which aims to generate realistic texts by unsupervised learning approach. for language gans, the diversity of the generated texts is an important evaluation metric. there are mainly two approaches to produce the diversity of texts by the generative models. one approach, which includes seqgan @xcite and leak-gan @xcite , is to generate the diverse texts by sampling words during the text-generation process. the generators of these models set the initial state as zero and randomly sample every word depending on the word distribution, and thus we cannot control the generated text depending on any conditions. the other approach, which includes textgan @xcite and fm-gan @xcite , produces the diversity of texts depending on the randomly sampled latent code from the prior distribution. these models set the latent code information at the initial state or the input of every time step for the generator. in this paper, we propose a gan model that aims to improve the approach to generating diverse texts from the latent space. as for textgan and fm-gan, the generator almost decisively selects each word using soft-argmax approximation to generate a sentence depending on the latent space information. to avoid mode-collapsing, instead of using standard gan objective function, the discriminator of each model respectively measures the maximum mean discrepancy (mmd) or the feature mover's distance (fmd) between the true text representations and fake ones. these models succeed in generating diverse texts if the generators are pre-trained by a variational autoencoder. however, it is verified that these methods still fall into mode-collapsing if the generator is not pre-trained (section 4.3.1). one of the possible reasons for the mode-collapsing is the deterministic word sampling process through a soft-argmax approximation from the beginning of the training. deterministic word sampling process hinders the generator from exploring a variety of text generation, which may lead the generator to fall into sub-optimal point. the second possible reason is that the discriminator tries to discriminate the completed sentences. generating a good-completed sentence from the beginning of the training is too difficult for the generator because the possible number of combinations of words increases exponentially if the number of words sampled becomes large. therefore, there is a possibility that, without pre-training, the discriminator does not serve useful signals to the generator if the discriminator looks at only completed sentences. based on these assumptions, our model adopts the following approach: the generator randomly samples words depending on the word probability distribution using the gumbel-softmax distribution @xcite . to ensure that the texts generated are conditioned upon the latent code, a reconstructor is introduced, which is fed the generated sentence and outputs the reconstructed latent code. the generator and the reconstructor cooperatively minimize the reconstruction loss. the discriminator of our model iteratively inspects incomplete partial texts and learns to distinguish whether they are real or fake using the standard gan objective. we trained our model using the coco image captions dataset @xcite for the experiment. the results show that although our model is not pre-trained, its performance is quite competitive with the existing baseline models. we also found that, by controlling the weight of the reconstruction loss coefficient, our model can obtain a higher diversity of generated texts even when texts are generated by greedy decoding. ## related work the language gans, in which the generator and discriminator optimize their objective functions in an adversarial manner to generate realistic texts, have two main perspectives. as the first perspective, a reinforcement learning approach is used for optimizing the generator. seqgan @xcite , leakgan @xcite , maskgan @xcite , rankgan @xcite , @xcite , and scratchgan (de masson d'autume et al., 2019) are typical models. these models are nondifferentiable from the discriminator to the generator. therefore, the generator cannot be optimized using a standard gan approach. instead, the output of the discriminator is regarded as a reward for the sampling of words, and the expected rewards are maximized to optimize the generator. this reinforcement approach generally produces the diversity of texts during the word sampling process. as the second perspective, the model is end-toend differentiable from the discriminator to the generator. textgan @xcite , relgan @xcite , and fm-gan @xcite are typical models. the sampling of words is ap- model generate text conditioned by latent space require pretraining seqgan, leakgan, etc no yes (mle) scratchgan, cot, mle no no textgan, fm-gan yes yes (vae) ours[gan], vae-based yes no other text generation approaches beyond those described above also exist, such as a vae-based model @xcite and cot @xcite to the best of our knowledge, our approach is the first gan model that does not require variational autoencoder pre-training and is able to generate texts conditioned by the latent code foot_0 . previous studies are summarized in ## model figure 1 shows a schematic illustration of the proposed method. we describe the details in the following paragraphs. our goal is to generate sentences conditioned by the latent space in a gan framework. when training language gans, if the discriminator only looks at the complete sentences, the generator obtains no learning signals early in the training be- the latent code z sampled from the prior distribution is fed into the fully connected layer and set as the initial state of the generator. the generator iteratively generates soft one-hot vectors of words using the gumbel-softmax distribution. the discriminator is fed the soft one-hot vectors and outputs the dense reward iteratively for the gan loss. the reconstructor is fed the vector-quantized hard one-hot vectors and outputs the reconstructed latent code for the reconstruction loss. this network is end-to-end differentiable from the discriminator/reconstructor to the generator. cause the complete sentences generated are easily determined to be fake by the discriminator(de masson d'autume et al., 2019). to address this problem, following the idea of fedus et al. (2018), semeniuta et al. (2018), and de masson d'autume et al. ( 2019 ), our discriminator d φ iteratively inspects incomplete partial texts and learns to distinguish whether they are real or fake. therefore, the generator can obtain more informative signals from the recurrent discriminator during the iterative word sampling process. by using this recurrent discriminator, it is expected that our model does not require pre-training, as reported in scratchgan (de masson d' @xcite . the objective function of the discriminator d φ is the same as that in scratchgan, except that the generator p θ generates a sequence of tokens {x 1 , ..., x t } depending on the sampled latent code z from the prior distribution p(z). in a practical sense, the typical word sampling process makes the differentiability from the discriminator to the generator impossible because sampling a word from a word probability distribution is equivalent to creating a non-differentiable onehot vector. as a workaround, we use the gumbel-softmax distribution @xcite , which enables our model to sample words from p θ by creating an approximated one-hot vector while making the differentiability from the discriminator to the generator possible. here, we call this one-hot vector a "soft one-hot vector." the gumbel-softmax distribution used to create the soft one-hot vector here, g is a randomly sampled value from the gumbel distribution gumbel(0, 1), o is the output from the generator. note that τ 1 is the gumbel-softmax temperature, and τ 2 is the word probability distribution temperature. to ensure that the texts generated are conditioned by the sampled latent code, our model introduces a reconstructor r ψ , which is fed the generated text and outputs a reconstructed latent code to minimize the reconstruction loss between the latent code and the reconstructed code. the generator p θ and reconstructor r ψ are optimized simultaneously. therefore, the objective function of the reconstructor is added to that of the generator multiplied by a coefficient λ. we found several heuristic approaches for stabilizing the training. as the first, vector quantization @xcite is applied to the soft one-hot vector to create a "hard one-hot vector" for the reconstructor input. vector quantization can back-propagate the loss from the reconstructor to the generator using only the sampled word parameters. by using this approach, reconstruction loss training is stabilized. figure 2 illustrates an example of the vector quantization process. as the second technique, if a blank token is chosen at any time step by the gumebel softmax of the generator, the rest of the sentence is automatically padded with blank tokens. the pseudocode is given in appendix c. ## experiment first, we describe the data setting and evaluation metrics for the experiment. second, we describe the experimental results to better evaluate the performance of our model. ## conclusion and discussion this paper proposed a gan model that aims to improve the approach to generating diverse texts conditioned by a latent space. in a quantitative experiment using the coco image captions dataset, it was shown that although our model is not pretrained, its performance is quite competitive with the existing baseline models, which require pretraining. future work will include further improvements to the performance of our model, and application of our model to other tasks that need to transform the data between domains through a latent space, such as improving the quality and diversity of machine translation or multi-modal learning related to text generation. real data a bicycle replica with a clock as the front wheel. a black honda motorcycle parked in front of a garage. a room with blue walls and a white sink and door. a car that seems to be parked illegally behind a legally parked car a large passenger airplane flying through the air. there is a gol plane taking off in a partly cloudy sky. blue and white color scheme in a small bathroom. this is a blue and white bathroom with a wall sink and a lifesaver on the wall. a blue boat themed bathroom with a life preserver on the wall vae a bathroom sink with only the tub in the bathroom a large boy and a plane sitting on the landing . a clock tower with pots and windows a car at an open door leading to a bunch of foot . office space force force jet on display during day . an image of benches on a street and chairs being terminal . a large airplane flying in the open of a kitchen . a couple of an airplane flying through the clear blue oven . an airplane with some chairs on a table by the counter . fm-gan a man wearing two sheep on a blue umbrella a group of birds standing around a table in a forest . a bathroom with a vanity , sink , and white and shower . a building with a clock on a clock tower a large white plane sits on a sidewalk in the kitchen . a row of cars are parked outside the street at an intersection . a woman looks plays in the kitchen an orange and woman walking around a park bench . a man standing in the kitchen at a tv . textgan a person with a football standing in front of a house an old airplane flying through a blue sky above a house . a man sitting on a bed with a dog and fries inside a car . a group of people riding bicycles down a city street . two motorcycles lined up with green seats in snow . a man wearing glasses wearing glasses and black bookbag riding a horse down a street . a bathroom with a toilet , shower , and toilet , trash can on the wall a cat drinking the back of a white toilet paper a man and motorcycle riders are riding on the road our model a small bird sit on a white bathroom with a mirror seat . two chefs counter standing in front of a toilet . a modern black and white checkered oven underneath area . looking off from you doors from doors . a white kitchen with chrome space at cabinets . a racing plane in a sky by land on a track . there is a yellow bathroom stands next to a toilet under a mirror . a kitchen with wooden appliances in flight a bathroom that has a mirror and a wall and basket . an image of men are crossing from the car .