NL -Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2023)

Abstract: Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training data for natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based natural language (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters (data splits according to specific features). We describe the framework and an initial set of 117 transformations and 23 filters for a variety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental human mistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguous to humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popular language models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases. The infrastructure, datacards, and robustness evaluation results are publicly available on GitHub for the benefit of researchers working on paraphrase generation, robustness analysis, and low-resource NLP.

## introduction data augmentation, the act of creating new datapoints by slightly modifying copies or creating synthetic data based on existing data, is an important component in the robustness evaluation of models in natural language processing (nlp) and in enhancing the diversity of their training data. most data augmentation techniques create examples through transformations of existing examples which are based on prior taskspecific knowledge @xcite . such transformations seek to disrupt model predictions or can be used as training candidates for improving regularization and denoising models, for example through consistency training @xcite . figure 1 illustrates a number of possible transformations for a sample sentence. however, the vast majority of transformations do not alter the structure of examples in drastic and meaningful ways, rendering them qualitatively less effective as potential training or test examples. moreover, different nlp tasks may benefit from transforming different linguistic properties. changing the word "happy" to "very happy" in an input is more relevant for sentiment analysis than for summarization. despite this, many transformations are universally useful, for example changing places to ones from different geographic regions, or changing names to those from different cultures. hence, a single repository that aggregates both task-specific and task-independent transformations will lower the barrier to entry for creating appropriate augmentation suites for any task. another advantage of supporting a broad range of transformations is the ability to capture the long-tailed nature and high diversity of surface forms of natural language @xcite . the current paradigm of testing models on data drawn i.i.d. from long-tailed distribution results in the head of the distribution being emphasized even in the test dataset and rare phenomena implicitly ignored by aggregate performance numbers. researchers have thus argued for more finegrained breakdowns of results in ways that capture these under-represented groups @xcite . however, the identification of these groups depends on and benefits from different cultural backgrounds and expertise. to capture a wide range of backgrounds, we thus capitalize on the "wisdom-of-researchers" and develop nl-augmenter in a participatory framework. nl-augmenter is a python-based natural language (nl) augmentation framework that aims to enable more diverse and be er characterized data during testing and training. 1 drawing upon researchers from computational linguistics, nlp, and other related fields, we collect 117 different ways to augment data for nl tasks. to encourage task-specific implementations, we link each transformation to a widely-used data format (e.g. text pair, a question-answer pair, etc.) along with the task types (e.g. entailment, tagging, etc.) that they support. nl-augmenter also provides more than 23 different filters, which can be used to create input subpopulations, according to features such as input complexity, input size, etc. unlike a transformation, the output of a filter is a boolean value, indicating whether the input meets the filter criterion, e.g., whether the input text is classified as toxic. we evaluate the robustness of four common pre-trained language models on four different tasks by testing their performance on perturbed test sets. the results demonstrate how nl-augmenter can easily corroborate prior findings that current pretrained models are strongly affected by small perturbations in texts. additionally, we expect nl-augmenter to be an effective tool for training data augmentation to develop models that are robust to diverse language characteristics. ## nl-augmenter nl-augmenter is a crowd-sourced suite to facilitate rapid augmentation of data for nlp tasks to assist in training and evaluating models. nl-augmenter was introduced in @xcite in the context of the creation of evaluation suites for the gem benchmark @xcite ; three types of evaluation sets were proposed: (i) transformations, i.e. original test sets are perturbed in different ways (e.g. backtranslation, introduction of typographical errors, etc.), (ii) subpopulations, i.e. test subsets filtered according to features such as input complexity, input size, etc.; and in this paper, we present a participant-driven repository for creating and testing transformations and filters, and for applying them to all dataset splits (training, development, evaluation) and to all nlp tasks (nlg, labeling, question answering, etc.). as shown by @xcite , applying filters and transformations to development/evaluation data splits allows for testing the robustness of models and for identifying possible biases; on the other hand, applying transformations and filters to training data (data augmentation) allows for possibly mitigating the detected robustness and bias issues @xcite @xcite . a majority of the augmentations that the framework supports are transformations of single sentences that aim to paraphrase these sentences in various ways. nl-augmenter loosens the definition of "transformations" from the logic-centric view of strict equivalence to the more descriptive view of linguistics, closely resembling bhagat and hovy (2013)'s "quasiparaphrases". we extend this to accommodate noise, intentional and accidental human mistakes, sociolinguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguous to humans @xcite . some transformations vary the socio-linguistic perspective permi ing a crucial source of variation wherein language goals span beyond conveying ideas and content. in this section, we provide organizational details, list the transformations and filters that the repository currently contains, and we present the list of tags we associated to transformations and filters and how we introduced them. ## robustness analysis all authors of the accepted perturbations were asked to provide the task performance scores for each of their respective transformations or filters. in section 4.1 we provide details on how the scores were obtained, and in section 4.2 we provide a first analysis of these scores. ## discussion and broader impact limitations in section 4.2, we analyze the results of applying some of the transformations on existing datasets and running models on the perturbed data. even though it was not possible to test all of the currently existing perturbations due to time constraints, the overall results show that the tested perturbations do pose a challenge to different models on different tasks, with quasi-systematic score drops. however, with so many transformations applied to four different datasets, the presented robustness analysis can only be shallow, and a separate analysis of each transformation would be needed in order to get more informative insights. second, our superficial analysis above relies on tags which were in many cases annotated by hand, and some of the surprising results (e.g. meaningpreserving are more challenging than non-meaningpreserving transformations) may reflect a lack of consistency in the annotations. we believe that assessing the quality of the tag assignment so as to ensure a high inter-annotator agreement will be needed for reliable analyses in the future. finally, the current robustness analysis only shows that the perturbations are effective for detecting a possible weakness in a model; further experiments are needed to demonstrate that the perturbations can also help mitigating the weaknesses they bring to light. ## conclusion in this paper, we introduced nl-augmenter, a framework for text transformations and filters with the goal of assisting in robustness testing and data augmentation tasks. we demonstrated that through an open participation strategy, nl-augmenter can cover a substantially wider set of languages, tasks, transformations, and filters than existing work, without a loss of focus. our repository @xmath0. we used these transformations to conduct robustness evaluations of popular transformer-based models and found that they are not robust, even to randomly (i.e., non-adversarially) sampled perturbations. although our analyses have revealed some aspects in which nl-augmenter can be improved, we showed how it can be beneficial to efforts in evaluating the robustness of nlp models. nl-augmenter can serve as a crucial resource for data augmentation especially for low-13 northern european journal of language technology resource domains and task-specific language processing. we welcome future contributions to improve its coverage of the augmentation space and to address its current shortcomings. investigating the effect on model robustness with larger-scale experiments is a potential direction for future work.