GDA : Grammar-based Data Augmentation for Text Classification using Slot Information (2023)

Abstract: Recent studies propose various data augmentation approaches to resolve the low-resource problem in natural language processing tasks. Data augmentation is a successful solution to this problem and recent strategies give variation on sentence structures to boost performance. However, these approaches can potentially lead to semantic errors and produce semantically noisy data due to the unregulated variation of sentence structures. In an effort to combat these semantic errors, we leverage slot information, the representation of the context of keywords from a sentence, and form a data augmentation strategy which we propose, called GDA. Our strategy employs algorithms that construct and manipulate rules of context-aware grammar, utilizing this slot information. The algorithms extract recurrent patterns by distinguishing words with slots and form the “rules of grammar”—a set of injective relations between a sentence’s semantics and its syntactical structure—to augment the dataset. The augmentation is done in an automated manner with the constructed rules and thus, GDA is explainable and reliable without any human intervention. We evaluate GDA with state-of-the-art data augmentation techniques, including those using pre-trained language models, and the result illustrates that GDA outperforms all other data augmentation methods by 19.38%. Extensive experiments show that GDA is an effective data augmentation strategy that incorporates word semantics for more accurate and diverse data.

## introduction text classification is a popular application of deep learning @xcite , and bert-based language models achieve a promising performance for the task @xcite @xcite . these popular language models, however, often exhibit a critical performance drop when the labeled data are insufficient. insert a song of bruno mars to bruno mars hits. language models often fail to learn the data distribution if the (sampled) dataset is too small to characterize the population @xcite . recent studies propose many techniques that alleviate the low-resource problem, such as data augmentation-a broadly accepted technique in the field of nlp. many data augmentation approaches such as eda @xcite and alp @xcite focus on syntactic variation to produce diverse data and have shown a definitive performance improvement. however, these strategies are not always reliable because they can lead to semantic errors as the variation might negatively impact the meaning of a sentence. we present some examples of sentences with logical errors in in text classification, a data sample pairs a nat-ural language sentence with its class, and a model learns to determine the classes from given sentences. several models utilize relations between the words to understand the given sentences, such as entities, part-of-speech (pos), and role semantics @xcite @xcite . we can use this information to augment the training data and improve the performance of language models. recent studies represent each sentence with a slot sequence in an effort to enhance sentence classification @xcite @xcite . these studies, however, show limitations on utilizing slot information because they only apply this information during the training phase of the models. we leverage slot information to enrich the diversity of generated sentences in data augmentation. given a sentence @xmath0, there is a sequence of slots @xmath1, which contains contextual information of words in s where @xmath2, and there is a class-label i of s, in which i represents the perceived intention of the sentence. see figure 1 for an example. while data augmentation in recent studies depends heavily on words present in the sentences, we focus more on the implicit properties of the words. for instance in figure 1 , the slot 'artist' indicates that 'iris dement' describes a name of an 'artist'. the other underlined phrase 'this is selena' could be understood as an individual sentence. however, the slot 'playlist' associated with the phrase implies that the phrase indicates a name of playlist, rather than a sentence by itself. these properties aid the model in comprehending such sentences precisely and act as a crucial component for interpreting the sentences. thus, we utilize these implicit properties to gain more appropriate data for each class. slots work by partitioning a sentence into meaningful word phrases. @xcite utilize the characteristics of slots in semi-supervised learning for pseudo-labeling procedures. their work infers a set of rules by substituting word phrases with symbols representing slots. after analysis, the authors conclude the approach is effective for low-resource problems. in a similar vein, we propose grammar-based data augmentation (gda) using slots. we first construct rules of grammar (rogs) by substituting keywords in the sentences with their coinciding slots. we then apply heuristics inspired by formal language to generalize these rogs. such manipulations provide variations in sentence structures and augment syntactically propitious sentences. we then produce a variety of sentences from the rogs and augment them to the training data. figure 2 describes the overview of gda. our definition of gda consists of three parts: rog construction, rule manipulation, and sentence generation. gda is a powerful data augmentation method when conducted on sentences with slots. it shows a maximum of 19.38% improvement in terms of prediction accuracy in the 5-shot experiment of sst-2. we also expand our method to more general types of datasets where slots are not initially available. with an intuition that named entities contain contextual information of the sentences @xcite , we conduct named entity recognition (ner) on the sentences and infer rogs by substituting words with their named entities. the experiment compares state-of-the-art data augmentation techniques to gda and demonstrates that gda is a successful implementation on typical text classification. ## related works data augmentation is a solution to the data scarcity problem which works by generating artificial data from the original dataset, with the key objective producing accurate and diverse data. traditional data augmentations involve synonym replacements or random word insertion, deletion, or substitution. more advanced augmentation techniques, such as symbolic augmentation, utilize alternative representations of natural language sentences @xcite . for example, eda @xcite applies predefined operations (rules) to modify sentences and @xcite proposes utilizing regular expressions to filter and clean noisy data while augmenting it. another line of research in data augmentation focuses on conditional generation using pre-trained language models-the results of which have been promising, especially given their significant performance improvements @xcite . large pre-trained language models like gpt-3 (brown et al., 2020) have gained popularity in data augmentation. however, the resource requirements for these models as well as the process of fine-tuning such large models for specific datasets is often infeasible and poses a challenge for individual users and researchers. recent studies explore the use of word properties from given sentences, such as slot information in task-specific datasets, like atis and snips, or named entities. in our work, we utilize this contextual information to overcome the challenges of employing large language models (llms) and we then propose a rule-based data augmentation approach based on context-free grammar. ## results and discussion we employ gda to augment datasets from section 4.1 and use the baseline classifiers from section 4.2 to evaluate the performance of augmentation. we illustrate the effectiveness of gda in tables 3 and 4 . we report scores of all combined strategies in appendices a and b. ## conclusions we have proposed a novel approach, gda which leverages the power of rogs and slot information and demonstrated how gda can effectively enhance data augmentation for text classification, particularly focusing on few-shot settings. gda solves a major issue among recent data augmentation techniques in which they potentially generate sentences with incorrect syntax or semantics, thereby limiting their effectiveness in improving performance. our experimental results have showcased the superiority of gda over the state-ofthe-art data augmentation methods, highlighting its performance gains. furthermore, we have conducted a qualitative analysis of the manipulation strategies inherent to gda, specifically examining the distance-and keyword-based manipulations. through these empirical experiments and analyses, we have confirmed that a hybrid approach combining both manipulation strategies yields the most effective results. our approach has provided insights on the utilization of slot information and rogs for data augmentation in text classification, particularly within the challenging few-shot settings.