GAN - LM : Generative Adversarial Network using Language Models for Downstream Applications (2023)

Abstract: In this work, we investigate Data Augmentation methods to improve the performance of state-of-the-art models for four different downstream tasks. Specifically, we propose Generative Adversarial Network using Language Models (GAN-LM) approach that combines a deep generative model with a pre-trained language model to produce diverse augmentations. We compare the GAN-LM to various conventional methods in non-contextual- and contextual-levels on four public datasets: ZESHEL for zero-shot entity linking, TREC for question classification, STS-B for sentence pairs semantic textual similarity (STS), and mSTS for multilingual sentence pairs STS. Additionally, we subsample these datasets to study the impact of such augmentations in low-resource settings where limited amounts of training data is available. Compared to the state-of-the-art methods in downstream tasks, we mostly achieve the best performance using GAN-LM approach. Finally, we investigate the way of combining the GAN-LM with other augmentation methods to complement our proposed approach. The developed code for reproducibility is included in the supplementary material.

## introduction nowadays, the availability of large unsupervised corpora and computational resources has led to development of large language models (lms) that are now employed across a wide variety of natural language processing (nlp) tasks including but not limited to entity linking (el), text summarization, question classification (qc) and semantic textual similarity (sts). while such models can sometimes work well for tasks where little (fewshot) or no (zero-shot) supervised data is available, the performance loss in such low-resource settings can be substantial compared to their highresource counterparts. this gap is even larger for low-resource languages. thus, scientists in both industry and academia still have to rely on a multitude of methods @xcite , such as data augmentation (da), to yield sufficient levels of performance on their low-resource tasks. da allows to artificially increase the size of a dataset by generating additional synthetic examples from the existing ones. a large amount of diverse training data is important to ensure the generalization of a model but it is not always possible to collect due to cost and time constraints or lack of target language data and task experts. to mitigate this issue, da can be used to improve performance. in this work, we test a wide variety of da approaches, both the ones found in the literature as well as our own approach, on four different tasks: zero-shot el with zeshel dataset, qc with trec database, sts with sts-b dataset, and multilingual sts with msts database. two different levels of augmentations are considered: (1) non-contextual, or word-level, and (2) contextual, where full sentence is considered for da. to further highlight the impact of different da approaches, we produced low-resource versions of the abovementioned tasks by subsampling training sets and removing rich textual contexts where applicable. we propose a novel generative adversarial network using language models (gan-lm) which employs gan with wasserstein distance to improve the stability of training and uses the pretrained lm for generating synthetic textual data to extend its usability. we also introduce tunable thresholds and a decoding method to control the diversity and lexical similarity of synthetic data to mitigate the mode collapse problem in gan. compared to other das, gan-lm employs an adversarial training with the offered data in each task to learn the characteristic of it which generates suitable synthetic data to aid in downstream tasks (covered in section 5.6). even if we used pre-trained lm in gan-lm, we do not use its generation ability (e.g. paraphrase, text generation) for downstream tasks. to complement our approach, we mix gan-lm with other das (e.g. back-translation, gpt) to enhance further in lowresource languages and limited entity linking task. the source code used to train the gan model and produce augmentations listed in this paper is publicly released and attached with a paper. ## related work originally, das for nlp relied on synonyms to increase diversity and dataset size. synonyms could be found in various resources like @xcite and @xcite . in wang and yang (2015), they considered word embedding with k-nearest-neighbor (knn) and cosine similarity to search and substitute similar words. other pre-trained word embeddings such as word2vec @xcite , glove @xcite and fasttext @xcite have been leveraged for that purpose. furthermore, the authors in @xcite generated synthetic texts by changing the words through synonym replacement or random insertions, substitutions and deletions where @xcite include the abstract meaning representation graph along with it for sts task. in @xcite , the authors simulated spelling mistakes by random insertions, substitutions, character swaps and deletions to enhance the robustness of the model for sentiment analysis. also, punctuation as da was considered in @xcite for qc task. later, back-translation with neural machine translation (nmt) was employed to generate variations of target words @xcite . more emerging techniques for da are using deep neural networks which mostly use autoregressive language model to predict words from a given context, e.g. gpt-2 radford et al. (2019 however, there are relatively few works using gans for text generation even if it is one of the most notable approaches in other domains @xcite . in kusner and hernández-lobato (2016), the authors used a gan model with gumbel-softmax to have a differentiable sampling distribution approximating a categorical one. in @xcite , diverse gans with recurrent and convolutional architectures were evaluated for text augmentation at word and characterlevels. @xcite proposed a sequence gan with reinforcement learning to address the problem of assessing a partially generated sequence. another work in @xcite developed a gan model consisting of relational memory-based generator, the gumbel-softmax relaxation, and multiple embedded representations in the discriminator. in golovneva and peris (2020), authors explored a data generation for the bootstrapping of a new language and the handling of low-resource features using a sequential gan. @xcite used the fine-tuning of bert with unlabeled data in a generative adversarial setting to reduce the time consuming of annotating the data but did not extend to the da application. similarly, @xcite use the cross-encoder to label the new inputs for training a bi-encoder model. @xcite focus on out-of-domain data generation with a sequential gan to build the robust dialog system. compared to these works, gan-lm combines a large pretrained model and gan with tunable thresholds to suitably control the diversity and similarity of generated data and it was tested on various downstream tasks to assess its generalizability. also, we can use any pre-trained lm on top of the gan part which extends its applicability to various tasks. to highlight the effectiveness of da, the low-resource settings are investigated in @xcite and @xcite where we mainly investigate the different size of training set and suggest a way to define the optimal size of augmented data. ## datasets and employed models we experiment with four different downstream tasks where (1-3) are english-based databases and (4) is multilingual-based dataset: (1) zeshel -a zero-shot learning dataset for el, (2) trec -a text retrieval dataset for qc, (3) sts-b -an integrated version of sts tasks for measuring the semantic similarity between two sentences, and (4) mstsmultilingual version of sts task. zeshel introduced by @xcite 2019 ) is based on wikia where there are non-overlapping domains in train/validation/test sets to simulate zero-shot learning. for this task, we employ blink's @xcite bi-encoder model from scratch. trec shown in @xcite is collected from @xcite where questions were manually created with 50 fine class labels. for this application, we use fine-tuned bert-tiny @xcite with training data in trec. sts-b covered in @xcite includes news headlines, image captions and user forum posts. in each sentence pair, semantic similarity labels are provided by a number between 0 and 5. for this task, we use sentencetransformers @xcite from scratch using the mean pooling layer with xlm-roberta @xcite . msts introduced in @xcite ; @xcite has sentence pairs in different languages with semantic similarity scores between 0 and 5. for train set, we used the offered monolingual pairs of ar-ar, es-es and the translated sentences of es-es into en, de, tr, fr, it, nl using google translator since we do not have monolingual pairs for them. the provided en-en dataset was eliminated from train set since most cross-lingual datasets were made from translating one sentence of en- @xcite . all the cross-lingual pairs are considered as test set. for this application, we employ the mean pooling of outputs for the pre-trained multilingual bert (mbert) @xcite with fine-tuning from train set. ## conclusions in this work, we investigate the effect of different das to improve the performance on various tasks. we study both techniques found in the literature as well as the proposed gan-lm in different scenarios: we subsample training sets to study model per-of the language resources and evaluation conference, reykjavik, iceland. european language resources association.