The Parrot Dilemma: Human-Labeled vs. LLM -augmented Data in Classification Tasks (2024)

Abstract: In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled data consistently exhibit superior or comparable performance compared to their synthetically augmented counterparts. Nevertheless, synthetic augmentation proves beneficial, particularly in improving performance on rare classes within multi-class tasks. Furthermore, we leverage GPT-4 and Llama-2 for zero-shot classification and find that, while they generally display strong performance, they often fall short when compared to specialized classifiers trained on moderately sized training sets.

## introduction large language models (llms), such as openai's @xcite , have demonstrated impressive zero-shot performance across a range of tasks, including code generation, composition of human-like text, and various types of text classification @xcite @xcite . however, llms are not perfect generalists as they often underperform traditional fine-tuning methods, especially in tasks involving commonsense and logical reasoning @xcite or concepts that go beyond their pre-training @xcite . additionally, the deployment of llms for downstream tasks is hindered either by their massive size or by the cost and legal limitations of proprietary apis. recently, competitive open-source alterna-tives such as llama @xcite , mistral @xcite , and falcon @xcite have emerged, allowing their use at a substantially lower cost compared to proprietary models. however, the training dataset sizes of these open-source models do not match those of their closed-source counterparts, and their performance across tasks remains somewhat uncertain. as an alternative to zero-shot approaches, researchers have explored the use of llms for annotating data that can be later used for training smaller, specialized models, thus reducing the notoriously high cost of manual annotation @xcite . previous work has primarily focused on using llms for zero-or few-shot annotation tasks, reporting that synthetic labels are often of higher quality and cheaper than human annotations @xcite . however, zeroshot annotations struggle with complex computational social science (css) concepts, exhibiting lower quality and reliability compared to human labelers @xcite @xcite . other work has proposed to mitigate these weaknesses by using llms to augment humangenerated training examples @xcite either through text completion of partial examples @xcite or through generation @xcite @xcite @xcite . research on data augmentation with llms is still in its early stages, exhibiting two main limitations. first, different classification experiments with synthetic augmentation produced mixed results; some demonstrated improvements in model performance @xcite while others observed minimal gains or even negative impacts @xcite . a recent review on the topic contributes to the assessment of an unclear landscape @xcite , highlighting that substantially smaller models fine-tuned on human-annotated data often outperform the llms. second, most previous work focuses on benchmarks that tend to be homogeneous in terms of their nature and complexity (e.g., sentiment classification), while disregarding more difficult or low-resource tasks. overall, the benefits of llms-based augmentation are not conclusive, especially when using them for training models for complex and low-resource classification tasks typical in computational social science (css) research. such prevailing uncertainty generates a dilemma of whether it is best to concentrate more resources into manual data labeling or into artificial augmentation. this work makes two contributions with the aim of bringing more clarity to this complex landscape. first, with the goal of providing css practitioners with a set of actionable guidelines for using llms in classification, we experiment with synthetic data augmentation on ten tasks of varying complexity typical of the domain of css. second, we perform a comparative analysis of strategies that incorporate llms into classification tasks either as data augmentation tools or as direct predictors. specifically, we assess how augmenting data with llmsgenerated examples performs compared to manual data annotation. we train our classifiers using incrementally larger datasets derived either from crowdsourced annotations or generated by gpt-4 or llama-2 70b, one of the best-performing opensource alternatives against closed-source model. we then contrast their performance to the zero-shot abilities of both the llms considered. overall, our work contributes to the current literature with three findings: ## methods we address ten classification tasks within the domain of css: (i) sentiment analysis @xcite , (ii) offensive language detection in danish (sigurbergsson and derczynski, 2023), (iii) extraction of social dimensions of language @xcite , (iv) emotions classification (crowdflower, 2016), (v) presence of empathy in text @xcite , (vi) identification of politeness @xcite , (vii) hyperbole retrieval @xcite , (viii) level of intimacy in online questions (pei and jurgens, 2020), (ix) whether two stances are at the same side of an argument @xcite , and (x) detection of condescension on social media posts @xcite . data for all tasks is publicly available. our experimental setup simulates a scenario where minimal manually labeled data is available, and additional labels are acquired either through human annotations or synthetic augmentation (figure 1 ). if test data is already available as separate from the training one in the original sources, we consider such a set as the test set. otherwise, we reserve 20% of the original data for testing. given the diverse sizes of the datasets and the time and economic constraints associated with using llms apis, we have set a threshold of 5, 000 samples to define the actual training set. we set aside a fixed base set of 10% samples from the actual training data, which we augment by generating 9 times the same amount of synthetic texts with gpt-4 and llama-2 70b chat ( §2.1). subsequently, we construct training sets of increasing sizes, starting from the base set and incrementing by 10% sample size either from the original data (crowdsourced dataset) or the synthetic data (augmented dataset), until reaching a maximum of 100% of the actual training data. for each dataset, we train a separate classifier ( §2.2), validate it on 10% randomly sampled data points from the actual training set for each training instance, and evaluate its performance on the holdout test set. to establish a baseline, we compare the trained models' performance with zero-shot classification using gpt-4 and llama-2 70b chat. we provide the models with a text and a set of possible labels, requesting them to classify the text accordingly (see appendix). we use identical prompts for both llms, with minimal changes to the template of llama-2 to align it with its pre-training format. all code and synthetically generated data are available on github foot_0 . ## results figure 2 illustrates the comparison between classification models trained on varying amounts of human-labeled and synthetically augmented data in terms of macro f1 score (results for other metrics can be found in supplementary and on w&b foot_1 ). three key findings emerge. first, models trained on human-annotated data generally outperform those trained on synthetically augmented data and zero-shot models in the cases of binary balanced tasks (cf. hyperbole), sensitive tasks (cf. condescension and offensiveness) and multiclass balanced tasks (cf. intimacy), even with limited sizes of training data. however, models trained on synthetically augmented data perform well on unbalanced multi-class tasks (cf. social dimensions and emotions), most likely due to the balanced data augmentation technique which substantially increases the number of samples for rare classes. in the specific case of emotions, the classification model based on llama-2 synthetically generated data outperforms all the other methods. synicon tags defined in second, zero-shot performance is strong only on specific tasks. for gpt-4, this holds particularly for sentiment, likely due to the vast amount of related data in gpt-4's training dataset, and same side stance tasks, possibly because of the small size of the test data available. gpt-4 also performs well in the second smallest dataset considered: politeness. in comparison, llama-2 performs substantially worse on sentiment, on-par on same side stance, and even better on politeness. for other tasks, the performance of zero-shot models is comparable to or even worse than that of classification models trained on either human-annotated or synthetically augmented data, particularly for intimacy and condescension. such tasks are characterized by a very nuanced difference between classes and by a notion of social "power" that cannot be extracted easily, given the complex paradigm of social pragmatics. a similar case of negative imposition of "power" is that of offensive, which is also characterized by a low zero-shot performance likely due to the restrictions of llms on offensive language. overall, only focusing on the zeroshot setting, we observe gpt-4 to be best on six tasks, equal in one task, and llama-2 best on three tasks. llama-2 was unable to produce any synthetically augmented text in danish for the task of offensiveness, thus we decided not to run the zero-shot llama classification for such a task. ## discussion and conclusion to enhance our limited understanding of the ability of llms to serve as substitutes or complements to human-generated labels in data annotation tasks, we investigate the effectiveness of generative data augmentation with llms on ten classification tasks with varying levels of complexity in the domain of computational social science. augmentation has minimal impact on classification performance for binary balanced tasks, but shows promising results in complex ones with multiple and rare classes. our findings lead to three key conclusions. first, the time to replace human annotators with llms has yet to come-manual annotation, despite its costliness @xcite , provides more valuable information during training for common binary and balanced tasks compared to the generation of synthetic data augmentations. second, artificial data augmentation can be valuable when encountering extremely rare classes in multi-class scenarios, as finding new examples in real-world data can be challenging. in such cases, our study shows that class-balancing llms-based augmentation can enhance the classification performance on rare classes. lastly, while zero-shot approaches are appealing due to their ability to achieve impressive perfor-mance without training, they are often beaten by or comparable to models trained on modest-sized training sets. overall, our study provides additional empirical evidence to inform the ongoing debate about the usefulness of llms as annotators and suggests guidelines for css practitioners facing classification tasks. to address the persistent inconsistency in results on llms' performance, we emphasize two essential requirements: (i) the establishment of a systematic approach for evaluating data quality in the context of llms-based data augmentation, particularly when using synthetic samples and (ii), the collaborative development of a standardized way of developing prompts to guide the generation of data using llms.