Controllable Data Augmentation for Few-Shot Text Mining with Chain-of-Thought Attribute Manipulation (2024)

Abstract: Prompting large language models (LLMs) for data augmentation has recently become a common practice in few-shot NLP tasks. In this paper, we propose Chain-of-Thought Attribute Manipulation (CoTAM), a novel approach that generates new data from existing examples by only tweaking in the user-provided, task-specific attribute, e.g., sentiment polarity or topic in movie reviews. Instead of conventional latent representation controlling, we leverage the chain-of-thought prompting to directly edit the text in three steps, (1) attribute decomposition, (2) manipulation proposal, and (3) sentence reconstruction. Extensive results on various tasks, such as text (pair) classification and aspect-based sentiment analysis, verify the superiority of CoTAM over other LLM-based augmentation methods with the same number of training examples for both fine-tuning and in-context learning. Remarkably, the 2D visualization of the augmented dataset using principle component analysis revealed a human-recognizable decision boundary that is likely hinted by the attribute manipulation, demonstrating the potential of our proposed approach.

## introduction prompting large language models (llms) for data augmentation has recently become a common practice in few-shot natural language processing (nlp) tasks. existing methods @xcite @xcite typically first generate new task-specific data with llms hinted by few-shot demonstrations and then fine-tune a (small) pre-trained language model with the augmented dataset for better performance. the same augmented data can be also incorporated into in-context learning (icl) @xcite . however, these augmentation methods usually prompt llms to generate new examples wildly without proper control, which hinders the informativeness of generated data and might induce spurious correlation. as shown in figure 1 (left), the generated data without control has no clear pattern and could even possibly mislead the fine-tuning or icl under few-shot supervision. in this paper, we propose a controllable data augmentation for few-shot text mining. the general idea is to generate new data from existing examples by only tweaking the user-provided, task-specific attribute, e.g., sentiment polarity or topic in movie reviews. intuitively, as shown in figure 1 , one can expect that this approach can efficiently find the decision boundary because we (1) directly manipulate along the direction of task-specific attributes and (2) maintain the rest of the attributes as before. different from the existing controllable generation works in computer vision @xcite and natural language processing @xcite , where attributes are manipulated in the latent space of the encoder before reconstructing new instances, we leverage the chain-of-thought (cot) prompting @xcite to directly edit the text using llms in three steps, (1) attribute decomposition, (2) manipulation proposal, and (3) sentence reconstruction. specifically, we start with the userprovided, task-specific attributes, and then prompt llms to decompose each individual text example into other orthogonal attributes. compared with a pre-defined attribute set per dataset, we believe that such dynamically constructed, per-example sets of attributes can better capture the uniqueness of every piece of text. second, we instruct llms to propose a plan to manipulate the values of the task-specific attributes while maintaining the other attribute values the same. finally, we prompt the llms to reconstruct the sentence based on the manipulation proposal. all these steps are written in a single prompt and fed to the llm at once. furthermore, using llms benefits the interpretability of our framework where attributes are completely transparent to users. we conduct extensive experiments to evaluate cotam and baselines using a series of few-shot classification tasks with very different classification targets, aspect-based sentiment analysis, and conditional text generation for more complex attribute manipulation. for a fair comparison, all compared methods utilize the same llms and generate the same amount of data. we assess the quality of generated data by looking at (1) the performance of trained small language models via fine-tuning or tuning-free methods on the augmented data and ( 2 ) the icl performance of llms using the augmented data as demonstrations. extensive experimental results including label-scarce and out-of-domain scenarios demonstrate the advantage of proposed controllable data augmentation over conventional methods. the ablation study further reveals the necessity of attribute manipulation comparing to directly flipping the labels. finally, we present pca analysis on the embeddings of generated augmentations that visually illustrates the effectiveness of method. our contributions are three-fold: ## problem formulation we aim to generate more efficient training data using controllable augmentation on a few-shot dataset d focusing on a target attribute y (e.g., the classification objective) with n possible values {y 1 , y 2 , • • • , y n } (i.e., n -way). for each possible attribute value y i , the dataset d provides k examples (i.e., k-shot) of texts with the value. we here showcase two mainstream few-shot learning schemes as the basis to discuss the augmentation: • in-context learning (icl) is a scheme for llms, which takes a few examples of sentences with their target attribute values (i.e., a series of (x, y i )) as the context to handle new inputs. with these demonstrations, the llm is expected to understand the underlying mapping and then predict the label of new inputs. ## our cotam framework to boost the performance of few-shot methods, we suppose a scenario, shown in figure 2 , to augment examples that well improve the task awareness of the inference models. for a given sample x with target attribute value y from d, we will manipulate its attribute value to y ′ that y ̸ = y ′ to form a build a new sentence x ′ . we set two requirements for the manipulation: 1) significant manipulation on the target attribute y , which means the manipulated result x ′ should be viewed with y j by oracle like humans. 2) minor manipulation on all other attributes z, which indicates x and x ′ to share a llm generates new samples with limited manipulation that is irrelevant to the task (i.e., controllable) user specifies the task and few-shot samples sentiment: negative sentiment: positive topic: movie structure: shift (1) attribute decomposition the other attributes and values? based on our desiderata above, we propose co-tam that benefits from the strong text manipulation capability of llms (openai, 2023) with its workflow demonstrated in figure 2 . to be more specific, we first create chain-of-thought (cot) queries to decompose the input texts into many attributes, which approximates the latent space. we aim to get rid of human labor to propose other possible attributes for efficiency. moreover, in some cases, even human experts cannot give you a complete list of other attributes among all the possible texts. finding a shared and fixed set of attributes for various kinds of texts is hard since different sentences rarely share a common set of applicable attributes. encouraged by @xcite , we instruct llms to propose a dynamic attribute set for each input text, which are customized among inputs dependent on which attributes are applicable. the cot then switches the value of the target attribute to other possible values in the task and prompts the llm to reconstruct the manipulated sentence. finally, the llm is guided to compose such a sentence to finalize the manipulation. different from the existing controllable generation works in computer vision @xcite and natural language processing @xcite , where attributes are manipulated in the latent space of the encoder before reconstructing new instances, our cotam is proposed to directly edit the text using llms. ## experiments in this section, we evaluate different llm-based augmentation methods on a series of classification tasks, with different target attributes. we incorporate comprehensive ways of utilizing augmentations with different classification techniques, such as fine-tuning, in-context learning and inference with sentence embedding. we further evaluate the augmentation ability of methods on more complex tasks like aspect-based sentiment analysis and conditional text generation. ## conclusion the study introduces a novel method, chain-of-thought attribute manipulation (cotam), which uses manipulated data from large language models (llms) for few-shot learning. our cotam creates label-switched data by modifying task-specific attributes and reconstructing new sentences. our testing validated the effectiveness of cotam over other llm-based text generation techniques. the results also showcase the potential for llm-guided learning with less supervision. future work will aim to adapt the attribute manipulation technique for smaller language models, increasing its accessibility. this would reduce reliance on the resource-intensive processes inherent to large language models, improving efficiency.