Task-driven augmented data evaluation (2022)

Abstract: In the area of data augmentation research, the main focus to date has been on the improvement of the generation models, while the examination and improvements to synthetic data evaluation methods remains less explored. In our work, we explore a number of sentence similarity measures in the context of data generation filtering, and evaluate their impact on the performance of the targeted Natural Language Understanding problem on the example of the intent classification and named entity recognition tasks. Our experiments on ATIS dataset show that the right choice of filtering technique can bring up to 33% in sentence accuracy improvement for targeted underrepresented intents.

## introduction recent advances in transfer learning methods have been a driving force in the progress of many natural language understanding (nlu) tasks. these methods typically involve pre-training of a largescale language model, followed by the task-specific fine-tuning @xcite . although these approaches have helped achieve state-of-the-art results on a variety of supervised learning tasks, they do not directly address the problem of task-specific annotated data sparsity. this is where data augmentation techniques come in to play, boosting model performance for a given supervised task by generating novel data points that are similar in characteristics to the available data. the main thrust of data augmentation research has been focused on improving generation models @xcite @xcite @xcite , while comparatively little work has been done on comprehensively evaluating and filtering high-quality synthetic utterances. current approaches suggest the use of a combination of automated metrics that evaluate utterances at * work done during the authors' tenure at amazon. popular word-based evaluation approaches are based on comparing n-grams in the original and generated text, and were originally developed for machine translation evaluations. among them commonly used scores are bilingual evaluation understudy (bleu) score @xcite , recall-oriented understudy for gisting evaluation (rouge) @xcite , and metric for evaluation of translation with explicit ordering (me-teor) @xcite . both bleu and rouge-n are based on comparing the n-gram overlap of the reference and generated texts and counts the number of token matches, with the difference being that rouge is recall-focused whereas bleu is precision-focused. meteor uses a set 1-gram mappings between the reference and generated text to get a weighted f-score, and adds a penalty function for incorrect word order. word-based sentence evaluation can give low scores for predictions with high lexical variation, but these predictions are not necessarily poor quality. to address that, one could use embedding similarity @xcite , that will measure the semantic similarity between the reference and prediction based on the cosine similarity between word and sentence embeddings. while high-quality generated data should be similar to the source data at hand, it is also important for this data to be novel. to measure the diversity of the generated text, one can use self-bleu @xcite score, which is computed by averaging the bleu scores of each generated utterance using the rest of the generated text as the reference set. furthermore, in an effort to address the diversity-quality trade-off of synthetically generated data, @xcite propose joint diversity-quality metrics: ms-jaccard similarity calculated as the average n-gram jaccard index of the generated text, and frechet bert distance (fbd) calculated as the frechet distance between the generated text distribution and the reference set distribution while using bert's feature space for the text. some approaches have been made to create a statistical model that would serve as an independent data evaluator. in their work, @xcite propose an rnn-based automatic dialogue evaluation model (adem) that predicts human evaluation score for the generated utterance. the drawback of this approach is that it requires additional data collection to train the model. despite a variety of approaches to annotated data evaluation, there is no golden standard, moreover, they often disagree on evaluation @xcite . however, the real value of the generated data can be only evaluated through the downstream task, for example by estimating how much performance improvement synthetic data can bring to the targeted supervised nlu task. in our work, we are connecting automated data evaluation with the downstream tasks, and apply it as a filtering mechanism to generated utterances. instead of attempting to build correlations with expensive human evaluations, we indirectly evaluate the quality of the generated data through the performance of the intent classification (ic) and named entity recognition (ner) tasks on the widely used air travel information system (atis) dataset @xcite . in summary, our contributions are as follows: ## synthetic data evaluation data sparsity is a common issue in multiple areas of nlu research. it is often a difficult and costly exercise for researchers to secure the required amounts of high-quality annotated data to train their models. in our work, we aim to generate synthetic data that can be used to improve nlu model training. we use the original training data along with intent and slot labels as a source to the data generation model. each utterance @xmath0, w 1 , ..., w n of length n is represented as a sequence of tokens, where w 0 is utterance's intent, and w i (@xmath1..n) denotes the ith slot of the utterance. in this paper, we will use sentence and utterance interchangeably. once data is generated, we use several text evaluation metrics to filter out high-quality utterances that will potentially bring greater benefit for the model. ## experiments in this section, we describe experimental setup, evaluation metrics and provide the summary of the experimental results.