Can the Transformer Be Used as a Drop-in Replacement for RNN s in Text-Generating GAN s? (2021)

Abstract: In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.

## introduction since the introduction of the transformer in late 2017 by @xcite , pure self-attention architectures have become ubiquitous in the natural language processing community. initially introduced by @xcite , the self-attention mechanism proved to be particularly useful in the context of neural machine translation to augment existing recurrent neural networks (rnns). recurrent neural networks (rnns), introduced by @xcite and expanded by @xcite ; @xcite , were the state of the art in learning and generating sequential models, notably for texts. with the addition of the attention mechanism, in the context of machine translation, they could focus better on the words and word combinations corresponding to the same concepts in different languages and focus on learning equivalences between them as opposed to trying to infer them directly from the whole excerpts of parallel texts used for learning. following the initial introduction of the attention mechanism to augment the rnns, architectures combining the two became de-facto state of the art. stacking rnn layers, adding pass-through mechanisms and separating the architecture into encoder and decoder with an attention layer in the middle became a standard, powering among others google's neural machine translation system @xcite . despite impressive performance, those architectures had a fundamental limitation to their ability to scale. the rnn training and evaluation are sequential by their nature, which means that architectures relying on the rnns could hardly benefit from the arrival of massively parallel computing. the innovation of the transformer was to show that it was possible to learn sequence-to-sequence mapping while dispensing entirely with rnn layers, using only self-attention mechanisms ("attention is all you need") - @xcite ). by stacking several layers of self-attention networks to form an encoder and a decoder, as well as introducing multi-head architectures, where each layer of self-attention network could be trained in parallel, the transformer and architectures derived from it scaled up easily and could be trained in parallel at a scale that was previously unreachable. that scalability enabled a continuous ramp-up of performance through parameter and training dataset size increase (brown et al., 2020), which eventually hurt itself against the limitations of reasonable demands on computational power in deployments. in turn, this led to an extensive research into making transformer-based architectures more efficient, focusing at first on specific instances @xcite and more recently on more general approaches (eg. @xcite ; @xcite al. (2020); fedus et al. (2021); ren et al. (2021) ). combined with the existence of specialized, energy-efficient hardware that is highly compatible with the transformer architectures, this make transformer-based architectures an attractive architecture to get the best value of limited computational budget. the text-generating capabilities of the transformer also gave rise to a new generation of models specialized in text generation. rather than mapping texts between languages, they focused on mapping a prompt to a text that would follow it. google's bert model @xcite isolated and scaled up the transformer encoder stack in order to perform masked training -predicting masked words in a text -referred to as autoencoding models @xcite @xcite . on the other side of the spectrum, openai's generative pre-training (gpt) family of models @xcite @xcite , focused on approximating the decoder stack and using the tokens from the prompt in order to initialize the hidden state of the self-attention modules stack -referred to as autoregressive models (by similarity with statistical autoregressive methods @xcite @xcite , see @xcite for machine learning applications). despite their impressive performance, when it comes to the text generation, both types of models are essentially autoregressive and are trained by max likelihood methods with regards to the training datasets. this poses several challenges. first, the autoregressive models learn the next token as a continuation of real texts they encounter in training, yet during the generation phase they continue from the text they themselves have generated. this means that during the generation phase they can rapidly go off the deeper end into an uncharted territory, they don't have a statistical model, and start generating degenerate output -a problem referred to as exposure biais @xcite . so far, solutions to this problem, such as scheduled sampling @xcite , are far from perfect and result in less diverse sampling and mode collapse @xcite . on top of that, the autoregressive nature of the model means that even in the territory where it has learned an appropriate token distribution, it will still be learning potentially undesirable biases @xcite , with no means to correct them other than to curate the entire dataset used to train the model (reviewed in depth by @xcite ). trying to learn the explicit statistical structure of natural language is, however, not the only way to train generative models. adversarial generative networks (gans) are a different training mode, where a generative model learns to generate outputs that are indistinguishable from the ones in the training dataset through a competition with a critic model, trained in tandem with it. introduced by @xcite , they are more robust to output degeneration, given that they always train in the generative mode, and require less computational resources than traditional auto regressive models. besides, a number of different pre-trained critics can be used to eliminate undesired biases or on the contrary, introduce desired ones, such as specifying an artistic style of an image @xcite . the adversarial learning approach has been highly successful for training image generation models, allowing high-quality image generation @xcite , day-to-night or summerto-winter image translation @xcite , or sketch-to-image translation @xcite . gans application to text generation, however, remained relatively limited. a major reason for that is that the sampling step leading to discrete and sequential token organization, needed for text generation, is problematic for gradient estimation, which is essential for training gans. as such, most existing text-generating gans rely on a max-likelihood autoregressive pre-training with the actual adversarial training phase being short and using a small learning rate, similar to a fine-tuning. unfortunately such an approach failed to address the shortcomings of the purely autoregressive models they acquire during the pre-training phase. however, recently de masson d' @xcite was able to demonstrate that it was possible to train a text-generating gan architecture from scratch, thus avoiding entirely the problems encountered by autoregressive max-likelihood methods. a curious property of the scratch-gan he developed, is that while it seems to solve problems that plagues both rnns and pure self-attention autoregressive models learning the explicit text token distribution, the authors still opted for rnn blocks, foregoing the advantages of transformer architectures in nlp and self-attention in image-generating gans @xcite . given the massive advantages of the transformer presents when it comes to training as well as with regards to the amount of research performed to make them more efficient, as well as better capabilities of transformers compared to gans, we wanted to know if it was possible to perform a drop-in replacement of rnns with transformers. such a transformer-based gan could provide two main benefits: the ability of produce higher quality samples at a reduced computational cost compared to traditional rnn-based gans, and a more scalable language gan. in order to approach this question, we chose to perform an experimental evaluation, based on a classical text-generating gan -diversity promoting gan (dpgan), developed by @xcite . we chose dpgan due to its straight-forwards architecture and training mode, similarity of its rewards structure to the state of the art text-generating gans de masson d' @xcite and the presence of the maximum likelihood pre-training, that we were expecting to be particularly favorable to the transformer layers. we refer to the dpgan with rnn layers replaced by a transformer as selfattention dpgan (sadpgan). ## related work the power of the transformer architecture in the language modelling tasks and its potential to further improve existing gan architectures did not escape the attention of the machine learning community. so far, self-attention architectures in gans focused on image generation tasks. perhaps the two best known examples are transgan @xcite and self-attention gan (sagan) @xcite . more recent advances, such as the introduction of generative adversarial transformers by @xcite have build on the transformer architecture even further, enabling long-range correlation to improve over the existing state of art, showing a potential for it in the gan setting. however, a common point between all of them is that they focus on the generation of images, rather than texts, and use only a single encoding layer for self attention or attempt to modify self-attention mechanism to better suite the image generation, diverging from the transformer architecture. the approach to text generation by combining the transformer and gans that comes closest to ours is the salsa-text, developed by @xcite . salsa-text is a textgenerating gan build around a transformer, discarding the original layer normalization and replacing it with the spectral layer normalization. in addition to that, salsa-text uses a modified transformer architecture, with less layers and a different structure, as well as a specific training regimen, meaning it is more of a gan built around a transformer rather than a gan where an rnn layer has been replaced with a pure self-attention based layer. here, we examine the transformer applicability as a general-purpose element that can be drop-in in architectures requiring a latent space encoding and thus directly replace rnns structures as lstms and gated recurrent units (grus). ## contribution and outline our contribution consisted in assessing the potential of transformers as a drop-in replacement of lstms in text-generating gans. through three different experiment involving sadpgan, we showed that transformers, despite achieving remarkable results in several nlp tasks, fail to adapt to the adversarial learning and adversarial fine-tuning context @xcite , causing sadpgan to consistently come short of the dp-gan performance and to exhibit severe mode collapse. this results suggest that as of now, the transformer can't be directly used as a replacement for lstm without further architectural and training mode changes. ## methodology our work is built upon an existing pytorch implementation of dpgan by @xcite . we used an iterative approach, first implementing a layer using exclusively transformer encoders, then adding the decoder stack with masking and finally adding teacher forcing during training. for each of these steps we compared the training results to the original implementation. since our goal was to investigate the possibility of replacing lstm layers in text-generating gans by dropping in transformer layers, and not to achieve new state of the art for text-generating gans, we kept our models relatively small: embedding and hidden dimension of 32, transformer encoder/decoder with two layers, each with 4 attention heads of size 64. the training loop for both gans consisted in 120 iterations of mle pre-training for the generator followed by another 120 epochs of adversarial training between the discriminator and generator. to assess the performance of both architectures, we used two negative log likelihood metrics, n ll gen and n ll div which measure the quality and respectively diversity of the generated text and (self)bleu scores when using real data. for the nll metrics lower values correspond to better results while for bleu higher scores are desirable. the code developed for this project is available from https://github. com/thebluehawk/ranlp21-70 , specifically the ranlp-2021 release. for the ease of use, it was packaged integrated into the code from @xcite . ## conclusion following our results, we have observed that a transformer architecture cannot be used as a simple drop-in replacement for rnns in the context of text-generating gans, at least not in its unmodified form. this result is not entirely surprising. despite a great performance of the transformer and its derivatives in different nlp tasks, transformerbased architectures are all but simple to train, as reviewed in @xcite . another problem with transformer architectures is their tendency to overfit the distribution of the text they are learning and fail to generate novel text in case insufficiently diverse and varied training datasets are used. in the case of the dpgan, the small size of the training dataset and the relatively small size of batches likely place us directly into the overfitting territory. upon the transition into the adversarial training, this overfitting is likely mutually amplified, leading to more and more degenerate text outputs. a more fundamental problem is that in dpgan, we are still faced with the transition from the maxlikelihood training regime to a generative regime. which means that even with improved datasets, the generator is likely to wander off into the uncharted territory as it tries to generate a new token based on the tokens it has already generated rather than tokes sampled from the training dataset. while this problem is present as well with the rnns, it seems that the output quality decay occurs faster with the transformer-based architecture. despite the existing wealth of text-generating gans, none, except for salsa-text @xcite are transformer-based, but persistently use rnns, notably lstm layers, including in the most recent, state-of-the-art ones, such as (de masson d' @xcite . in case of salsa-text, a modified architecture and a different regularization methods are used, both in two specific setups. it seems that part of this trend in using rnns rather than a transformer-based architecture is rooted in the existence of fundamental differences in the way the transformer learns compared to the rnns, that make it particularly vulnerable to the hazards of adversarial training regiments. as a result, we expect that developing text gan architectures using self-attention based architectures instead of rnns ones would require designing new gan architecture from scratch to ensure a initialization and evaluation/reward structure that would be compatible with the the transformer layer. a pure self-attention layer capable to be a dropin replacement for rnns is still to be developed. given the similarity between the adversarial phase occurring in the majority of current text-generating gans and the adversarial fine-tuning mechanism (reviewed by eg. jeddi et al. ( 2020 )), we expect that text-generating model fine-tuning to avoid undesirable patterns and adversarial prompts (such as exemplified in @xcite ) would not be straightforwards and rely on the development of more robust and stable self-attention architectures.