Data Augmentation using LLM s: Data Perspectives, Learning Paradigms and Challenges (2024)

Abstract: In the rapidly evolving field of large language models (LLMs), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of LLMs on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From both data and learning perspectives, we examine various strategies that utilize LLMs for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for diverse forms of further training. Additionally, this paper highlights the primary open challenges faced in this domain, ranging from controllable data augmentation to multi-modal data augmentation. This survey highlights a paradigm shift introduced by LLMs in DA, and aims to serve as a comprehensive guide for researchers and practitioners.

## introduction data-centric approaches to artificial intelligence (ai) constitute a pivotal element in the advancement towards artificial general intelligence (agi), centering on the construction of ai systems underpinned by high-quality data @xcite . this emphasis on data quality is vital, as it ensures the clarity of the information from which ai systems are to learn. nevertheless, the acquisition of high-quality data presents significant challenges, being both costly and time-intensive, while the data annotation phase is often laborious and prone to inaccuracies resulting from human involvement @xcite . in response to these challenges, researchers have dedicated efforts towards data augmentation (da) techniques as a means to mitigate such issues @xcite . data augmentation fundamentally involves the adoption * equal contribution, order decided by coin flip. as we venture into the realm of large language models (llms), the significance of data concerns escalates. research into the scaling laws pertinent to llms highlights the critical role of data as a renewable resource crucial for the enhancement and advancement of models @xcite . with the expansion of model training scales, there is a marked increase in data consumption. prior studies have examined the escalation in dataset sizes within the domain of machine learning, forecasting that the reservoir of high-quality linguistic data may be depleted by 2026 @xcite . this anticipates a potential deceleration in the growth of machine learning models unless there is an enhancement in data efficiency or the discovery of novel data sources. consequently, the utilization of synthetic data produced by ai models becomes essential once high-quality human-generated data resources are fully exploited. from the data perspectives, data augmentation using llms offers a viable strategy to overcome these limitations, facilitating the creation of synthetic datasets of high quality that can, in certain instances, exceed the value of data curated by humans @xcite . this strategy not only addresses the challenge posed by the limited supply of human-annotated data but also conforms to scaling laws, enabling an increase in the size of training datasets without a proportional escalation in computational costs (flops) @xcite . the tactical employment of synthetic data stands to substantially reduce data collection costs and energy usage, signifying a transformative phase in model training and inference and laying the groundwork for the achievement of artificial general intelligence (agi) @xcite . from a learning perspective, data augmentation using llms has heralded innovative learning paradigms, marking a significant departure from traditional methods predominantly centered around tasks like machine translation, sentiment analysis and ner. the application of llms in data augmentation extends to a broader spectrum of learning paradigms, including instruction tuning, in-context learning, and alignment learning. additionally, it facilitates the generation of pseudo data for classification purposes and the scoring of data for regression analysis. this evolution in methodology not only broadens the scope of potential applications but also invites a deeper exploration into the nuanced benefits and insights that such innovative paradigms offer in the realm of data augmentation. as such, the systematic review of these emerging paradigms is crucial for advancing our understanding and leveraging the full potential of data augmentation in these learning contexts. given the growing interest and work in this domain, we believe it is a timely moment to present a paper on llm-based da. this paper aims to: (i) discuss data augmentation using llm from the data perspective (ii) explore the learning paradigms that involve training llms on data generated by llms themselves, and (iii) highlight the principal challenges in this field to effectively guide and spur further interest and research. to the best of our knowledge, this represents the first survey that delves into data augmentation methods leveraging llms in such detail, thereby marking a significant contribution to the literature on llm applications. this paper is structured as follows. section 2 provides a comprehensive review of related surveys, highlighting the distinctions between this work and previous surveys. following that, as illustrated in figure 1 , section 3 presents analysis from data perspectives for data augmentation with llms, establishing a foundational understanding for the community. section 4 explores the learning paradigms associated with data augmentation using llms, neatly categorizing existing methods into two primary types: generative learning and discriminative learning. section 5 ventures into the challenges and prospective future directions for research in this domain, highlighting the complexities and opportunities that lie ahead. to further aid the community, the appendix offers a detailed listing of existing methods categorized by tasks (section a) and domains (section b), serving as a valuable resource for researchers and practitioners alike. through this survey, we hope to attract wider attention, generate increased interest, and encourage further research in the field of data augmentation using llms. ## data perspectives from a data perspective, we group existing studies on llm-based da into four categories: 1. data creation which leverages the few-shot learning ability of llms to create a large synthetic dataset; 2. data labeling which uses the llm to label existing datasets; 3. data reformation which transforms existing data to produce new data; 4. co-annotate which enables llm-human collaboration to gather high-quality augmentation data. this section discusses relevant papers in each category. ## learning paradigms teacher-student learning (tsl) leveraging a language model (llm) as a data annotator to create synthetic data for the purpose of training other models represents a pivotal shift towards more efficient and scalable machine learning methodologies. this approach is situated within the broader framework of teacher-student learning (tsl) @xcite , where the llm acts as the 'teacher' by generating annotated data that serves as training material for 'student' models as shown in figure 2 . by utilizing an llm for data annotation, organizations can bypass some of the limitations associated with human annotators, such as scalability issues and subjectivity, thereby streamlining the development process of ai systems. this innovative strategy not only enhances the efficiency of generating training datasets but also contributes to the advancement of machine learning techniques, enabling the creation of more robust and accurate models. this emerging trend is gaining traction, particularly in the context of training large language models. teacher-student learning paradigms for data augmentation using llms can be broadly segmented into generative and discriminative learning approaches. generative learning exploits llms to create instructional datasets and demonstration examples, enriching model training. it encompasses supervised instruction learning, generating instructional data (instruction, x, y); in-context learning, producing few-shot examples to augment query inputs and facilitate output generation; and alignment learning, creating human-preference data for reinforcement learning and optimization. discriminative learning, conversely, focuses on refining task-specific models through pseudolabeling and pseudo-scoring. pseudo-labeling generates labels (x, ŷ) for smaller models, akin to knowledge distillation, while pseudo-scoring assigns numerical scores to (x, y) pairs for regression tasks. ## conclusion in this survey, we have delivered an in-depth and organized review of data augmentation techniques utilizing large language models (llms). we explored key data augmentation strategies and the application of llms in enhancing data augmentation processes. moreover, we discussed the learning paradigm under the generative and discriminative learning perspectives. additionally, we highlighted the existing challenges and potential future research opportunities, indicating significant potential for further developments in this field. our aim is for this paper to act as a reference for ai researchers, aiding them in selecting appropriate data augmentation methods and encouraging further investigation and enthusiasm in this domain.