Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis (2023)

Abstract: Generative Adversarial Networks (GAN) is a model for data synthesis, which creates plausible data through the competition of generator and discriminator. Although GAN application to image synthesis is extensively studied, it has inherent limitations to natural language generation. Because natural language is composed of discrete tokens, a generator has difficulty updating its gradient through backpropagation; therefore, most text-GAN studies generate sentences starting with a random token based on a reward system. Thus, the generators of previous studies are pre-trained in an autoregressive way before adversarial training, causing data memorization that synthesized sentences reproduce the training data. In this paper, we synthesize sentences using a framework similar to the original GAN. More specifically, we propose Text Embedding Space Generative Adversarial Networks (TESGAN) which generate continuous text embedding spaces instead of discrete tokens to solve the gradient backpropagation problem. Furthermore, TESGAN conducts unsupervised learning which does not directly refer to the text of the training data to overcome the data memorization issue. By adopting this novel method, TESGAN can synthesize new sentences, showing the potential of unsupervised learning for text synthesis. We expect to see extended research combining Large Language Models with a new perspective of viewing text as an continuous space.

## introduction generative adversarial network (gan), as proposed by @xcite , is a popular model for data synthesis. gan is an unconditional data generation algorithm that aims to generate plausible data in an unsupervised manner by fostering competition between a generator and a discriminator to capture the real data distribution. when gan was initially introduced, it primarily focused on image synthesis, and extensive research was conducted to achieve high-quality synthetic data results @xcite @xcite . furthermore, gan is commonly employed in the field of computer vision for data augmentation through image synthesis @xcite @xcite . the gan generator learns implicit density based on the discriminator's loss without direct reference to the training data. consequently, gan can prevent data memorization, where the model reproduces the training data. additionally, gan can synthesize various data by using random noise instead of a specific starting point, such as a designated start token. similar to images, unconditional text generation can function as a data augmentation technique by generating new text that resembles a given dataset. it also has practical applications, such as creating new documents by generating fictitious text information suitable for direct use. consequently, several studies have attempted to apply gan to natural language, but they have encountered limitations in natural language generation. the challenge arises from the fact that natural language is composed of discrete tokens, making it challenging for the gan generator to directly update gradients through backpropagation. the gradient backpropagation issue in text-based gans was first discussed by @xcite , and numerous subsequent text-gan research efforts aimed to address this problem using gradient policy-based reinforcement learning with a reward system. furthermore, the previous text-gan approaches necessitated pre-training the generator with supervised learning (autoregressive) before adversarial training due to convergence issue with the generator @xcite . accordingly, we discovered that the generators of previous text-gan approaches reproduce the training data (leading to data memorization) during text synthesis due to the autoregressivebased pre-training process, which becomes a significant text embedding space gan for text synthesis issue in generative models. this paper introduces a novel framework known as text embedding space generative adversarial networks (tesgan) foot_0 , which enables backpropagation and prevents data memorization. tesgan does not rely on a supervised, pre-trained autoregressive-based generator that generates discrete tokens for text synthesis. our generator generates continuous text embedding spaces for text synthesis instead of discrete tokens, allowing training with gradient backpropagation. furthermore, the fact that tesgan deals with continuous spaces makes it possible for tesgan's generator to be trained within the original gan framework to mimic the real text embedding space. moreover, tesgan enables unconditional text generation, as it does not require the selection of a starting token (or prompt) for text synthesis. our seed interpretation model then synthesizes sentences by interpreting the imitated continuous text embedding space created by the generator. during sentence synthesis, data memorization does not occur because tesgan does not directly refer to the training text data but only learns from the continuous text embedding space. we use two datasets to conduct performance evaluations and general applicability experiments based on synthetic text generated by tesgan. to assess the quality and diversity of synthesized text, we employ evaluation metrics such as fréchet bert distance (fbd), multi-sets-jaccard (msj) @xcite , language model score (lm) (de masson d' @xcite , and self-bleu (sbl) @xcite . in addition, we conducted human evaluations, and tes-gan achieved the highest average score. lastly, we calculate the data memorization ratio and present the synthesized sentences to assess the potential of unsupervised learning and continuous embedding spaces for text synthesis. ## related works the most common method of text generation is to use an autoregressive-based language model via teacher forcing @xcite . for example, extensive studies have been conducted on models using recurrent neural network (rnn) with long short-term memory (lstm) cells @xcite . using lstm, @xcite successfully generated handwriting by predicting sequences, and @xcite synthesized sentences under specific conditions. @xcite generated text after learning text embedding spaces with an autoregressivebased lstm model and a variational autoencoder (vae) architecture @xcite . policy gradi-ent with bleu (pg-bleu) calculates the bleu @xcite score of synthesized sentences and takes them as a reward when updating the generator using policy gradient. numerous investigations have been conducted to utilize gans for text synthesis. sequence gan (seq-gan) @xcite attempted to address the backpropagation problem by employing gradient policybased reinforcement learning with a reward system. however, seqgan faced a reward sparsity issue, leading @xcite to introduce rankgan, which replaced the previous regression-based discriminator with a novel ranker. rankgan trains the discriminator to assign higher scores to more realistic sentences. maskgan @xcite utilized an lstm-based generator to fill in masked parts of sentences with tokens during training. since maskgan uses discrete tokens, gradient backpropagation is not possible for the generator. to overcome this challenge, the authors employed the actor-critic method, using the probabilities of candidate tokens from the discriminator as rewards during training. @xcite proposed maximum likelihood augmented discrete gan (mali-gan), which synthesizes text by minimizing kullback-leibler divergence @xcite . leak-gan @xcite alleviated issues related to sparseness and the lack of intermediate information by providing leaked information from the discriminator. several studies have aimed to address the gradient backpropagation problem without relying on rewardbased reinforcement learning. textgan @xcite introduced kernel-based moment-matching, which enforces empirical distributions of real and synthetic text by using lstm and convolutional neural networks (cnn) for the generator and the discriminator, respectively. feature mover gan (fm-gan) @xcite defined the feature-mover's distance (fmd) and learned it by minimizing the fmd between real and fake sentences. both textgan and fm-gan utilized lstm generators that generate discrete tokens using the soft-argmax trick instead of relying on reinforcement learning. relational gan (relgan) @xcite applied relational recurrent neural networks @xcite and attempted to address the gradient backpropagation issue using gumbel-softmax @xcite . however, since these approaches employed autoregressive (e.g., lstm) generators, they explicitly referenced the training text data during model training. consequently, previous studies faced challenges in avoiding complete data memorization while synthesizing sentences due to an autoregressive generator. lastly, transformer-based implicit latent gan (tilgan) @xcite adopted a similar approach to tesgan for addressing the gradient backpropagation issue based on the embedding space. however, tilgan differs from tesgan in that it was trained on a latent space compressed by the encoder, configured as an autoencoder transformer, and did not utilize embeddings learned from real language models. most of the aforementioned text-gan models require the first token or prompt to synthesize text due to their autoregressive generators. tesgan stands apart from these models as it generates text embedding space directly from random noise, eliminating the need for selecting tokens. our tesgan is the first text-gan model that learns the real text embedding space without relying on an autoregressive generator. ## text embedding space gan tesgan aims to generate the seeds required for synthesizing plausible text. these generated seeds (fake seeds) from the generator, along with the real seeds from the real text, are passed to the discriminators for training within the gan framework. once the training of tesgan is complete, the pre-trained seed interpretation model synthesizes text using the fake seed created by the generator. ## conclusion in this work, we proposed a novel unsupervised text synthesis framework, tesgan. tesgan facilitated the gradient backpropagation of natural language discrete tokens by creating a continuous text embedding space called a seed. in most text-gan studies, data memorization had been inevitable because the generator had to be pre-trained with an autoregressive approach be-fore adversarial training. therefore, we introduced tes-gan, which mitigated the data memorization issue by applying an unsupervised gan framework that does not directly refer to the training data. tesgan improved text synthesis performance during adversarial training and resulted in the best or comparable results in terms of evaluation metrics. additionally, tesgan exhibited the lowest data memorization ratio, and the data memorization study confirmed that these results were attributable to the tesgan framework. furthermore, tesgan achieved the highest scores in human evaluations. the ablation study highlighted the importance of the four objective functions, and the synthetic text results from a large dataset-trained tes-gan demonstrated its general applicability. this paper underscores the potential of continuous embedding spaces in conjunction with discrete tokens for text synthesis through unsupervised learning. by integrating the concept of viewing text as a continuous space with publicly available large language models (touvron et al., 2023), models can synthesize more expressive sentences, and we anticipate that many follow-up studies will emerge.