Building Adaptive Acceptability Classifiers for Neural NLG (2021)

Abstract: We propose a novel framework to train models to classify acceptability of responses generated by natural language generation (NLG) models, improving upon existing sentence transformation and model-based approaches. An NLG response is considered acceptable if it is both semantically correct and grammatical. We don’t make use of any human references making the classifiers suitable for runtime deployment. Training data for the classifiers is obtained using a 2-stage approach of first generating synthetic data using a combination of existing and new model-based approaches followed by a novel validation framework to filter and sort the synthetic data into acceptable and unacceptable classes. Our 2-stage approach adapts to a wide range of data representations and does not require additional data beyond what the NLG models are trained on. It is also independent of the underlying NLG model architecture, and is able to generate more realistic samples close to the distribution of the NLG model-generated responses. We present results on 5 datasets (WebNLG, Cleaned E2E, ViGGO, Alarm, and Weather) with varying data representations. We compare our framework with existing techniques that involve synthetic data generation using simple sentence transformations and/or model-based techniques, and show that building acceptability classifiers using data that resembles the generation model outputs followed by a validation framework outperforms the existing techniques, achieving state-of-the-art results. We also show that our techniques can be used in few-shot settings using self-training.

## introduction a key component of these models is a synthetic error generation step that applies various sentence * work done while on leave from ohio state university. ## related work work on automated evaluation metrics in the tradition of bleu @xcite shares similar goals as our work, except that such metrics make use of reference sentences and thus are not designed for use at inference time. moreover, such methods have not been found to correlate well with human evaluation of individual texts outside of the machine translation paradigm @xcite . @xcite presents a comprehensive literature survey of the three broad categories of evaluation of the text generation models-human, automated and machine-learned-along with providing strong motivation for doing nlg evaluation. 1 our approach is inspired by work in the third category of machine-learned evaluation. as noted, @xcite improve upon earlier heuristic-based filtering by generating synthetic error data for training a semantic fidelity classifier. to do so, they use simple sentence transformations to create artificial omission, repetition, hallucination and value errors. however, since such transformations are not adaptive to nlg generation models the classifier is used with, they may not always produce the kind of unacceptable samples the corresponding nlg model would. also related is @xcite work on building a machine-learned scorer, bleurt, to replace automated metrics such as bleu. they use mask filling with a pretrained language model for creating synthetic unacceptable examples. in this paper, we introduce several new techniques for synthetic data generation, and comprehensively evaluate them in comparison to @xcite 's methods, as well as to bleu and bleurt. in addition, we introduce a validation framework to sort the samples into the 2 classes. our validation framework uses a pretrained entailment model, similarly to how @xcite use one for semantic evaluation; here, we go beyond their approach by using it to develop an adaptive acceptability classifier that is better suited to runtime use. as an alternative to using acceptability classifiers, one can make use of reconstruction models @xcite to determine how well the nlg model's output predicts its input. these models are capable of detecting content errors but are not designed to capture grammatical mistakes. additionally, since such approaches employ a second autoregressive decoding step, they are less well-suited to runtime inference in systems with tight latency budgets. regarding our self-training experiments, we note that self-training has been previously investigated for nlg by @xcite , @xcite and stevens-guille et al. (2020), though they do not explore using pre-trained models with self-training. also related are earlier approaches that use cycle consistency between parsing and generation models for automatic data cleaning @xcite . more recently, @xcite have developed a method for randomly generating new text samples with gpt-2 then automatically pairing them with data samples. by comparison, we take a much more direct and traditional approach to generating new text samples from unpaired inputs in self-training @xcite , using pre-trained models fine-tuned on the few-shot data for both generation and reconstruction filtering. ## datasets we conducted experiments on 5 datasets: webnlg foot_1 @xcite , cleaned e2e @xcite , viggo @xcite , a conversational weather dataset created following the method described in @xcite , and an alarm dataset released by @xcite . further, in order to examine the effectiveness of self-training for building the acceptability classifier in the few-shot setting, we delexicalized @xcite the meaning representations in the webnlg dataset. for nlg models fed as inputs to our framework, we built an lstm model for alarm and finetuned bart @xcite ) models for others as per arun et al.'s recommendations. for test sets, we used (1) system outputs from the public human evaluation set for webnlg 2017 and converted the labels to acceptable class if both grammar and semantics ratings were greater than 2 (out of 3); (2) the data sabotaging strategy described in section 5.1 to create model responses for cleaned e2e, @xcite (3) responses generated by the weather and alarm nlg models for those datasets. we used these methods as a practical way to create a variety of errors in sufficient quantities to be able to effectively test the acceptability classifiers. additionally, the weather and alarm test sets are representative of current sota models built for these domains. human evaluations were done for all test sets except webnlg to determine acceptability, using two annotators and a tie-breaker round in case of disagreement. the number of samples in all human annotated test sets can be found in ## framework design in figure 1 , we show the overall design of our proposed framework. the framework takes in as input the training data of the text generation model as well as the trained generation model. the next step is the synthetic data generation that makes use of these inputs and is able to generate as many samples as needed. the synthetic samples are then passed through a validation framework that either sorts them into acceptable and unacceptable classes or rejects them altogether. ## synthetic data generation our synthetic data generation methods use the training data of the generation models (seed data) and the trained generation model. in sections 9 and 10, we observe that a classifier built on data using our model-based and adaptive approaches improves upon the average f1 scores of standalone non-adaptive approaches by 1.1% to 18%. following are the 4 strategies we introduce. ## validation framework the validation framework takes in the synthetic samples generated by above described methods and filters and sorts them into acceptable and unacceptable classes. our experiments in section 9 show that using a validation framework improves macro f1 scores across all models by 1.4% to 5%. following are the techniques we introduce. ## classifier model architecture we formulate the task as binary classification with labels {acceptable, unacceptable} and learn a discriminator model using both the training data of the generator models and synthetic samples generated and refined using our adaptive approach. the training data consists of generation model input concatenated to the response (synthetic or original text) with a separator token. for the underlying model architecture, we use roberta-base @xcite . ## few-shot setting recently, there have been several efforts to train nlg models in a few-shot setting @xcite @xcite . we adopt the self-training strategy introduced by @xcite to generate training data for generative models, which is also used as the seed data needed for acceptability modeling. self-training consists of several cycles of generation and reconstruction. for generation, we fine-tune bart @xcite using only 500 annotated examples to generate nlg responses given the input meaning representations (mrs). the same generation data is used to finetune a reconstruction bart model to obtain the input mr given the response. we use the reconstruction model to select samples with the exact reconstruction match after the generation step. at the end of the self-training cycles, we use all the selected samples as seed data for acceptability modeling. following @xcite , we delexicalize the meaning representations of the webnlg dataset and pair them with existing delexicalized responses. ## results in the following subsections, we compare the overall performance of classifiers. we report precision (p) and recall (r) of both acceptable (a) and unacceptable (u) classes. we also report macro-f1 (f1) as the main metric we use to compare performance of the generation techniques. we use 10-fold cross validation (cv) to adjust the classification thresholds used for making predictions on the test sets. to calculate standard deviation of the f1 values, we use bootstrapping with 1000 rounds. finally, we report the means over 3 runs for each technique. further, we perform mcnemar's statistical significance test comparing models trained with the best combination of methods with those trained with sentence transformations. we use the following abbreviations to refer to the different synthetic data generation techniques: sentence transformation (snt), data sabotaging (sbtg), noisy beam search (nbm), mask filling with vanilla bart (bart), mask filling with finetuned bart (ftb). we performed nearest neighbor analysis (using bleu) between test and synthetic unacceptable responses generated during training by our adaptive methods and sentence transformation. we found that across datasets, 52.27% to 98.48% unacceptable responses have a closest match to a sample generated by an adaptive method, suggesting that adaptive techniques produce more realistic samples compared to sentence transformation. in ## conclusion in this paper, we introduced and analyzed several model-based and model-adaptive techniques, along with a validation framework, to create synthetic acceptable and unacceptable responses for training acceptability classifiers to filter outputs of neural nlg models. in addition, we compared and contrasted combinations of these techniques with using only the simple sentence transformation methods recently introduced by @xcite . we carried out a comprehensive study using 5 nlg datasets with varying levels of complexity and demonstrated that a combination of our methods and sentence transformations deliver state-of-theart performance on all of them. additionally, we demonstrated that using self-training, our models can be trained in few-shot settings without any significant drop in performance. this is especially important in light of recent efforts to develop few-shot nlg models, where avoiding semantic errors remains a central challenge. finally, we recommend the strategy of using fine-tuned bart with the entailment model validator for building an acceptability classifier in the absence of a representative validation set. when such a set is available, we recommend performing ablation experiments across all combinations of different techniques using our framework in section 4. further analyzing the various conditions under which different synthetic data generation and validation strategies work with respect to the nature of underlying data is left for future work.