C o D a: Constrained Generation based Data Augmentation for Low-Resource NLP (2024)

Abstract: We present CoDa (**Co**nstrained Generation based **Da**ta Augmentation), a controllable, effective, and *training-free* data augmentation technique for low-resource (data-scarce) NLP. Our approach is based on prompting off-the-shelf instruction-following Large Language Models (LLMs) for generating text that satisfies a set of constraints. Precisely, we extract a set of simple constraints from every instance in the low-resource dataset and verbalize them to prompt an LLM to generate novel and diverse training instances. Our findings reveal that synthetic data that follows simple constraints in the downstream dataset act as highly effective augmentations, and CoDa can achieve this without intricate decoding-time constrained generation techniques or fine-tuning with complex algorithms that eventually make the model biased toward the small number of training instances. Additionally, CoDa is the first framework that provides users explicit control over the augmentation generation process, thereby also allowing easy adaptation to several domains. We demonstrate the effectiveness of CoDa across 11 datasets spanning 3 tasks and 3 low-resource settings. CoDa outperforms all our baselines, qualitatively and quantitatively, with improvements of 0.12%-7.19%. Code is available.

## introduction data augmentation is a widely used technique to address the problem of limited training data in lowresource nlp @xcite . owing to the recent advancement in generative ai, using synthetic data to train task-specific models has also gained much popularity. however, given a low-resource nlu dataset, effectively generating task-specific data to expand the dataset still poses a significant challenge. for example, while diversity in tokens and contexts in generated augmentations typically 1 https://github.com/sreyan88/coda * these authors contributed equally to this work. method original 1: scott gimple sets the record straight on that big midseason finale scene. original 2: mine is all of isaiah chapter 6 because that is what jesus did for me. backtrans 1. scott gimple is setting the record on this. (yu et al., 2018) 2: all of isaiah, chapter 6, is mine, for this is what jesus. eda 1: scott gimple sets the record straight on that big midseason finale scenery. (wei and zou, 2019) 2: mine is all of isaiah chapter because personify that is what jesus did for me. ssmba 1. scott gimple sets the record, on that big preseason night website. (ng et al., 2020) 2: it in all of isaiah -6 because that is what you did for me. gpt3mix 1: scott gimple sets the record straight on that big midseason finale scene. (yoo et al., 2021) 2: i cherish isaiah chapter 6 as it embodies what jesus has done for me. genius 1: for the record, there was no midseason finale scene. that is, until the finale. @xcite 2: isaiah chapter 6 because it says, "if you don't believe in god, you will die." in today's society, we must continue to mine the teachings of jesus to find the courage to live our lives on our own terms. extracting constraints . . . llm 1 2 constructing the instruction augmentation gemeration 3 fixed instruction ## related work generative data augmentation for low-resource nlp has been extensively studied in prior work and can be categorized into four primary techniques. firstly, text-infilling involves corrupting source text segments and using a plm to refill these gaps. this process often relies on conditioning the corrupted text, a concept also known as keyword conditioning in some studies @xcite @xcite . secondly, text editing focuses on modifying certain parts of a given sentence @xcite . thirdly, prompting involves generating new training sentences by prompting llms @xcite , which can be further uncategorized into conditioning attributes, exemplars, or constraints derived from training data. ## methodology fig. 1 illustrates the coda pipeline. given a lowresource dataset @xmath0, ⋯, d i , ⋯d n }, we first extract a set of simple heuristic-based constraints from each document d i and then verbalize the constraints to construct an instruction i d i . after this, we either instruct the llm with i d i to generate a completely new document or rephrase another existing document from d. for the latter, we first retrieve a document from d, convert it into its short and concise abstract description by prompting an llm, and then employ i d i to generate a document from the abstract description and the extracted constraints. for retrieval, we calculate cosine similarity between sentencebert embeddings @xcite of the source document d i and all other documents in d, and we randomly sample a sentence from the top-k and bottom-k similar sentences. for a total of 5 augmentations, we generated 3 novel documents and rephrased 2 other documents for every d i . finally, all the generated augmentations are added to d for training a downstream nlu model. we now describe our methodology to extract constraints in detail. ## experimental setup baselines. gold-only refers to training our model only on the low-resource gold data. for sequence classification (sc), we compare coda with text editing baselines: eda @xcite , aeda @xcite , and amr-da @xcite , learning-based infilling baselines: ssmba @xcite , genius(-ft) @xcite , promda @xcite , llmbased prompting baselines: zerogen @xcite , @xcite and rephrasing baselines: backtrans @xcite . for the intent classification task, specifically in sc, we add another llm-based prompting baseline: promptmix @xcite . for named entity recognition (ner), we compare coda with lwtr @xcite , daga @xcite , melm @xcite , promda @xcite and aclm @xcite . finally, for question answering (qa), we compare it with zerogen, backtrans, genius, eda, and aeda. details on the working of all baselines are provided in section d. finally, for qa, we employ squad @xcite and newsqa @xcite . details on each dataset and dataset statistics are provided in section c. hyper-parameter settings. we prompt llama-13b with a temperature of 0.5, top-p of 1.0, top-@xmath1. for all downstream nlu tasks, we employ bert base-uncased @xcite as our encoder (except ots where we employ legallongformer large @xcite ). we finetuned our encoder with a batch size of 4,8 for 100 and 200 splits and 16 for 500 and 1000 splits. for ner specifically, we employ the flair library (ak-bik et al., 2019) with an initial lr of 1e -5 and constant decay. appendix a provides hyper-parameter tuning experiments. we report the micro-average f 1 score averaged across 3 runs for 3 random seeds. ## results and analysis quantitative analysis. ## conclusion we present coda, a simple and controllable data augmentation technique for low-resource nlp. coda extracts simple heuristic-based constraints from source sentences and verbalizes them to construct and instruction, which is then used to prompt llms to generate augmentations. coda is trainingfree and works with any out-of-the-box instructiontuned llm. beyond providing explicit control, coda is also flexible, i.e., constraints can be easily replaced or added, enhancing its suitability across diverse domains.