Is C hat GPT the ultimate Data Augmentation Algorithm? (2023)

Abstract: In the aftermath of GPT-3.5, commonly known as ChatGPT, research have attempted to assess its capacity for lowering annotation cost, either by doing zero-shot learning, generating new data, or replacing human annotators. Some studies have also investigated its use for data augmentation (DA), but only in limited contexts, which still leaves the question of how ChatGPT performs compared to state-of-the-art algorithms. In this paper, we use ChatGPT to create new data both with paraphrasing and with zero-shot generation, and compare it to seven other algorithms. We show that while ChatGPT performs exceptionally well on some simpler data, it overall does not perform better than the other algorithms, yet demands a much larger implication from the practitioner due to the ChatGPT often refusing to answer due to sensitive content in the datasets.

## introduction textual data augmentation (da) is a rich and complicated field, with the goal of finding ways to generate the most informative artificial examples to add to a training set without additional labelling cost. many techniques have been developed and thoroughly tested in order to create informative artificial data. recently, the new model chatgpt has put into question much of what was known for dataset construction, with researchers even wondering if its capacities meant the end of human labelling @xcite . while extensive studies have been done on the use of chatgpt for many natural language understanding tasks, its use for da has been surprisingly little investigated yet. to our knowledge, only two papers have looked into this, namely @xcite and @xcite . both however explore limited settings, which makes it still unclear whether chatgpt is a good tool for data augmentation or not. in particular, @xcite study the performance of chatgpt and gpt-4 on classification tasks of medium size (500 examples), comparing zero shot data generation and few-shot+data augmentation to crowdsourcing annotation, but not to other da techniques. @xcite compare the use of da with chatgpt for large datasets (several thousand examples), focussing on bio-medical data. their algorithm however does not isolate chatgpt for data augmentation, but instead combines it with pre-training, preventing an objective evaluation of the capacities of chatgpt for da. in this paper, we compare the use of chatgpt for paraphrasing as well as for zero-shot data generation, on five classification datasets (three binary and two multiclass ones), to seven algorithms that have shown good performances in the past. we show that the performance of chatgpt is highly dependent on the dataset, due mainly to poorly defined tasks, which makes prompting difficult. these tasks were chosen because they are standard on the textual da literature, and as such these biases are important to point out. with efficient prompting, generating new data with chatgpt remains the best way to perform textual da. ## related work there are roughly three main types of da techniques: word-level augmentation, paraphrase, and generative methods. 1 in word-level da, operations modify genuine sentences' words to create variations. commonly, the operation is word substitution, replacing it with a synonym @xcite , a neighboring word in pre-trained embedding @xcite , or by masking and predicting with a neural network @xcite @xcite . paraphrasing techniques attempt to create paraphrases from the available sentences. the most seminal technique of this family is back-translation (bt), a technique in which a sentence is translated to a pivot language and then back into english @xcite @xcite @xcite . neural networks to directly generate paraphrases have also been used, with specialized decoding techniques for rnn @xcite , or by using a bart model trained on a corpus of paraphrases generated from bt @xcite . generative methods learn the distribution of the training data and generate new data from it. while the obvious advantage is that data should be more diverse, generative models are often more complicated to train and fine-tune correctly. examples of this family of methods includes using gpt-2 for generating new data @xcite @xcite , other generative models such as vaes @xcite @xcite or conditional vaes to generate examples conditionned on the class @xcite @xcite . finally, there have also been interest into the use of proprietary models for da. both yoo et al. (2021) and @xcite show that gpt-3 is able to generate excellent new data, either by completing a list of sentences from one class, or by asking to generate both new sentences and their labels. chatgpt has also been studied to generate new data, asking it to paraphrase existing data @xcite for few-shot learning, but the authors first fine-tune the classifier with a large dataset from the same distribution, making it hard to isolate the impact of the generated sentences. finally, møller et al. (2023) look at the performance of chatgpt and gpt-4 for data augmentation compared to human annotation, and conclude that for simple datasets (such as review analysis of products), chatgpt is better, but otherwise human annotation outperforms data generated by chatgpt. as they do not compare to other da techniques, it is also hard to know how chatgpt performs. ## algorithms it is not clear from the literature which da algorithms perform best, and so in order to thoroughly test the capacities of chatgpt we select a variety of techniques to compare da objectively: eda, aeda, cbert, cbart, cgpt, bt, t5-tapaco, chatgpt-par and chatgpt-desc. we briefly describe each of those algorithms, and refer to the code for the full details of the implementations and hyper-parameters. 2 eda and aeda are two simple word-level algorithms that achieved great performances in the past. in eda, one of four operations is chosen (insertion of related words, swapping words, deleting words, and replacing words by synonyms) and applied to a percentage of the words in the sentence. 3 in aeda @xcite , punctuations are randomly inserted in the sentence (among "?", ".", ";", ":", "!", and ","), the number of insertion being randint(1, len(sentence)/3). cbert and cbart have very similar methodologies. we prepend the class of the example to all genuine sentences, mask a fraction of the tokens, and fine-tune the model on the available training set to predict the masked words. for generation, we then give the modified sentence (masked and with the class prepended) and pass it through the transformers. the main difference between cbert and cbart is that the latter can predict spans instead of tokens, which allows more flexibility. cgpt also works by prepending the class to the sentence, which then allows gpt-2 to learn to generate conditionally to it. for generation, we give the class as well as the separator token and let gpt-2 generate new sentences. 4 in bt, we first translate the sentence to a pivot language and retranslate it in english, creating paraphrases. we use the fsmt model from hugging face foot_5 , with the intermediary language being german, which has been shown to obtain good performances @xcite . @xcite propose to fine-tune bart on a corpus of in-domain paraphrases created with bt. we found in our experiments that we could get results just as good by using t5-small-tapaco 6 , which is the t5 model fine-tuned on the corpus of paraphrases tapaco @xcite . finally, we test the use chatgpt, either by asking for paraphrases of genuine sentences (chatgpt-par), or by giving a short description of the task and classes and asking for novel sentences (chatgpt-desc). we give the exact prompts in appendix a. because part of the experiments were done before the api became publicly available, we use in this paper the pro version of the web interface of chat-gpt and leave further fine-tuning for future work 7 . ## datasets and methodology we test on five datasets with various complexity and characteristic to fully assess the performance of the algorithms. we use sst-2 @xcite , a binary dataset of movie reviews classification, fakenews foot_8 , a dataset of news to classify into real news or fake ones, irony and ironyb @xcite , a binary and multiclass version of a task consisting into classifying tweets as ironic or not, and which kind of irony for the multiclass version (polarity clash, situational irony, other irony), and trec6 @xcite , a multiclass dataset where the goal is to classify questions into six categories (abbreviation, description, entities, human beings, locations, and numeric values). more information is available in appendix c. these datasets were chosen to get a spread of tasks, and because they are commonly used in the literature in data augmentation. sst-2 and trec6 are both fairly standard in da research, being used for example in @xcite @xcite . the irony datasets are also used quite regularly, for example in @xcite @xcite . finally, while fakenews has not been used in da to our knowledge, it is still commonly used for fake news detection, for example in @xcite @xcite . we test data augmentation on two settings: few shot learning (10 or 20 starting examples), and classification with dataset sizes of 500 and 1000 examples. while sampling the starting set, we make sure to balance the classes to be able to observe the performance of data augmentation without the additional factor of imbalanced data. we also tested the process on the full dataset but, similarly to other papers, we found da to have little effect in this setting. as such we do not discuss them in the paper, but we include them in appendix c. based on results reported in the literature as well as our experiments, we use a larger ratio of generated-to-genuine sentences for small datasets and a smaller one for larger datasets. concretely, we generate 10 new sentences per genuine sentences for the dataset sizes of 10 and 20, and one for 500 and 1000. 9 as a classifier, we use bert, and we fine-tune it using the development set, using early-stopping. we run each experiment 15 times, reporting accuracy for binary tasks, and macro-f1 for multiclass tasks. ## results for small dataset sizes, we observe that the performance of chatgpt-par is on par with the best algorithms, but doesn't beat them by a significant margin on average. while chatgpt-desc performs exceptionally well, its performance comes almost exclusively from sst-2, for which it generates highly informative data. for other datasets, it mostly either brings no gain in the performance or a very small one. overall, all the algorithms provide a reasonable augmentation of the performance, except maybe cgpt, which performs poorly. excluding chatgpt-desc, bart and t5-tapaco obtain some of the best performance, although not by much. given the generative nature of chatgpt-desc and its performance, one could also wonder if it would perform better if we generated more sentences. in appendix c we show that this is not the case, and that the performance for all datasets plateau quickly. for larger training sets, chatgpt performs better, while bart and t5-tapaco degrade the performance. we believe that this is due to these algorithms creating paraphrases which are closer to the original sentences, leading to less diversity overall. while on few-shot learning this is not a problem because the goal is to give the neural network enough relevant data to learn, on larger datasets diversity seems to become a prevalent factor. nevertheless, the overall effect of augmentation on moderately .5-par 88.2/89.1 73.8/77.7 66.8/69.3 42.8/45.9 82.4/87.1 70.8/73.8 gpt3.5-desc 87.4/88.9 71.9/75.9 64.1/66.9 41.1/44.4 79.8/84.0 68.9/72.0 sized datasets are very small, which brings the question of whether data augmentation is relevant at all in these cases. ## discussion of all the algorithms used here, t5 and chatgpt present the greatest novelty as well as show some of the best performances. as such, we center our discussion on these two algorithms. when we observe t5 sentences (see appendix b), we can see that they are not as good as one would expect, often being ungrammatical or badly formed. still, it has been noted before that having correctly formed sentences is not an important criterion for the efficiency of a da algorithm, which might explain why its performance is high @xcite . chatgpt-desc often has difficulty generating sentences of the desired class, accounting for its overall poor performance, while chatgpt-par creates excellent paraphrases, bringing diversity to the dataset while maintaining class coherence. nevertheless, there is a hidden cost that we found was not discussed in other papers, namely the need for data reparation. chatgpt-par quite often refuses to create paraphrases, especially for the fakenews and irony/ironyb datasets which contain occasional mentions of rape and sex. in these cases, we had to manually find the "bad" element of the batch and rerun it without it, adding considerable complexity to the data augmentation algorithm. another option would be to simply not correct them, but our preliminary studies indicate that this degrades the performance of da significantly. foot_10 6.1 poorly defined tasks and dataset biases. despite the description strategy being able to add something akin to external data, our experiments show that chatgpt underperforms with this method, the performance often being worse than when paraphrasing the existing data. this raises many questions as adding more diverse data should augment performance. we found that for most part, the poor performance of chatgpt was related to the poor health of the datasets. except for sst-2, we found that fakenews, irony, ironyb, and trec6 have poorly defined labels in relation to the task, and that examples in the datasets were often ambiguous to human eyes. under these conditions, it is difficult to expect chatgpt to perform well. we underline these problems here because poor dataset health is not a rare phenomenon. irony and ironyb are two datasets of the se-meval 2018 competition. data was collected from twitter by collecting tweets that contained some specific hashtag such as #irony, #not, or #sarcasm, which were then removed to form the data of the ironic class. the non-ironic class was then formed by collecting other tweets. this creates a heavy bias in the dataset which shift the task from predicting if the tweet is ironic to predicting if there was a #irony hashtag coming with it. without the clue that the hashtags give us, it is often impossible to know if the tweet is ironic or not, and we show in appendix c some examples of ironic tweets which, from manual labelling, we found were ambiguous in their classes. trec6 is a dataset of the text retrieval conference and consists in classifying the questions into six categories. while all data was manually annotated, we found inconsistencies in the annotation. for example, "what is narcolepsy?" is labelled as description but "what is a fear of motion?" as an entity. other inconsistencies are "what is the oldest profession?" and "what team did baseball 's st. louis browns become?" labelled as human vs "what do you call a professional map drawer" as entity, or "where did indian pudding come from?" being labelled as description but "where does chocolate come from?" as location. given that the same mislabelling remains in the test set (ex "where does dew come from?" being labelled as location), chatgpt generating sentences of the correct class won't help the classifier much. it is to note that these issues were already noted by @xcite , who advise using multilabel classification to reduce biases introduced in the classifier. in all of its usage for da however, we found it used as a regular classification problem, with all the ambiguity problems it entails. finally, fakenews is a kaggle dataset which has been thereafter used in many papers for fake news detection. we decided to use this dataset because it seemed a difficult and interesting task, but while analyzing it, we found it biased in a sense similar to irony and ironyb. from what little information we could find, news were gathered from various sources and split into real or fake news based on the journal they came from. this causes biases because while some journals may have a tendency to sprout fake news, it does not mean all of its news are fake. furthermore, we found strange choices of labelling. for example, all articles from breitbart are labelled as real news even if it receives a mixed score of factual reporting foot_11 and articles from consortium news, which receives the overall same score, are labelled as fake 12 . by refining prompting, we can augment the trec6 dataset to go to 68.6, which still underperforms when compared to bert foot_13 . we found chat-gpt to have difficulty understanding the concept of "entity" and "human" questions, often labelling them instead as "description". ## conclusion data augmentation is a seminal technique to lower annotation cost and keep good performances, but even today it is difficult to figure out which technique works best. in particular, the use of chatgpt has not been correctly assessed for data augmentation, leading to the unknown factor for industries of whether it is worth the price. in this paper, we study nine data augmentation techniques, including a novel one using a pretrained t5 system for paraphrasing, and show that while chatgpt achieves among the best results, it doesn't outperform the other algorithm by a significant margin. this, coupled with the fact that using chatgpt costs both time and money when compared to the other algorithms, brings us to a different conclusion than what previous studies using chatgpt for da found, namely that it might not be worth it depending on the task. we further found that while zero-shot generation of data could give outstanding results, it was often hindered by biased datasets, which prevented efficient prompting of chatgpt.