Improving D utch Vaccine Hesitancy Monitoring via Multi-Label Data Augmentation with GPT -3.5 (2023)

Abstract: In this paper, we leverage the GPT-3.5 language model both using the Chat-GPT API interface and the GPT-3.5 API interface to generate realistic examples of anti-vaccination tweets in Dutch with the aim of augmenting an imbalanced multi-label vaccine hesitancy argumentation classification dataset. In line with previous research, we devise a prompt that, on the one hand, instructs the model to generate realistic examples based on the gold standard dataset and, on the other hand, to assign multiple pseudo-labels (or a single pseudo-label) to the generated instances. We then augment our gold standard data with the generated examples and evaluate the impact thereof in a cross-validation setting with several state-of-the-art Dutch large language models. This augmentation technique predominantly shows improvements in F1 for classifying underrepresented classes while increasing the overall recall, paired with a slight decrease in precision for more common classes. Furthermore, we examine how well the synthetic data generalises to human data in the classification task. To our knowledge, we are the first to utilise Chat-GPT and GPT-3.5 for augmenting a Dutch multi-label dataset classification task.

## introduction in many text classification settings, the label distribution in datasets is imbalanced, which impacts the learning process of machine learning models and often leads to a degraded performance @xcite . this phenomenon is especially inherent to multi-label datasets and therefore complicates multi-label classification further. multiple approaches for tackling classification tasks where the data is imbalanced have been proposed. these approaches, which aim to alleviate the issue of class imbalance, comprise multiple categories and can be divided into data re-sampling, classifier adaptation and ensemble methods @xcite . @xcite describe multiple data augmentation (da) methods, which encompass da of the text itself at character, word, sentence/paragraph, or document level. these methods include noise induction by replacing and swapping characters or words, embedding replacement, replacing words with synonyms, translation, or even a combination of multiple methods. with the popularisation of large generative models such as the gpt models @xcite , researchers have leveraged them for da purposes, ranging from augmenting text translation datasets @xcite , to generating examples for an event detection dataset @xcite and augmenting a covid-19 q&a dataset @xcite . however, most research is restricted to binary or multi-class classification, thus leaving much uncovered ground to research the potential benefits of augmenting multi-label data. additionally, the cross-lingual capabilities of the state-of-the-art (sota) generative models for da purposes remain under-explored. in this paper, we augment a dutch dataset for vaccine hesitancy argumentation monitoring, as introduced in @xcite using sota language models, including chat-gpt, and build upon previous research by introducing these generative models in a realistic multi-label text classification setting. additionally, we contextualise the performance with a strong back-translation baseline and extensively compare different gpt-based da methods for multi-label text classification. our contributions are the following: the present work is organised as follows: section 2 provides an overview of research into data augmentation and the most recent generative models. section 3 describes the datasets that are used for the experiments, including how the generative models were prompted, introduces the classification models used and outlines the evaluation methods. in section 4, the results from the experiments are summarised and additional extensive analyses are conducted. lastly, section 5 briefly summarises the presented methodology and findings, along with outlining steps for future work. ## methodology in this work, we aim to build upon existing research by augmenting a dutch multi-label vaccine hesitancy detection dataset using novel generative models. given the capabilities of the most recent models, we aim to generate realistic examples of vaccine-hesitant tweets to alleviate the data scarcity issue of the present application. we hypothesise that introducing new synthetic examples in the training data will improve the performance of large language models on the mentioned down-stream classification task. ## results and discussion in this section, the results for the experiments are summarised. first, the effect of data augmentation will be discussed. then, in order to gain more insight into the nature of the synthetic data and the generative models, additional sets of experiments were conducted. first, we perform a typicality study on the synthetic and human data. then, we directly compare the data generated by chat-gpt and the gpt-3.5 api by balancing the data and utilizing the balanced sets for the classification tasks in order to determine if there is a clear difference in quality between the two synthetic datasets. additionally, we measure how consistent the generative models are in labeling the generated instances by training and testing robbert on the synthetic data only. furthermore, we investigate how well the synthetic data generalises to human-written data by training models on the synthetic data only and testing on the cross-validation test sets. finally, we evaluate the impact of the synthetic data in a cross-platform setting where the models are trained on twitter data and tested on facebook comments. ## conclusion in this paper, we leveraged two instances of the gpt-3.5 model for augmenting a dutch multi-label anti-vaccination dataset. using these models for generating data and adding pseudo-labels, we evaluated the impact of this augmentation method by comparing the performance of multiple language models on a multi-label downstream classification task. we show that the present method can lead to a significant performance increase, especially for underrepresented classes, thus highlighting the potential of the present approach for other applications. moreover, we demonstrate that the class distribution of the generated datasets depends on the model instance that is used and the number of gold standard examples in the prompt itself. we compared the performance of the proposed method to back-translation, a strong baseline method for performing da that achieves a roughly equal performance to gpt-based augmentation. furthermore, we found that a combination of back-translation and gpt-based augmentation was the most effective for our application. with research into gpt models still continuing, future work should extend this method to other multi-label tasks, as data scarcity and data imbalance are inherent to this classification problem. additionally, while the gpt models have the potential to majorly accelerate progress in the field of ai, the inner workings of the gpt models lack transparency, thus undermining attempts at versioning and replicating results. we therefore believe that providing powerful, openaccess models are vital to progress in the field.