Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference (2020)

Abstract: A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text, i.e. checking if the output text contains all and only facts supported by the input data. We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI). We use the NLI model to check textual entailment between the input data and the output text in both directions, allowing us to reveal omissions or hallucinations. Input data are converted to text for NLI using trivial templates. Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.

## introduction neural models may reduce the effort for building natural language generation (nlg) systems and produce very natural outputs, at the cost of limited control over the model outputs. state-of-the-art neural d2t models are prone to omitting or hallucinating facts @xcite @xcite , which restricts their real-world deployment. recognizing these errors is thus essential for proper system evaluation and further research in d2t generation. in general, evaluating the semantic accuracy of d2t generation outputs requires full natural language understanding. minor changes in wording may cause major differences in the meaning of the text, making it difficult for handcrafted heuristics to cover all edge cases. human evaluation, on the other hand, is expensive and difficult to scale. we note that the task of checking if a generated sentence includes/entails a particular fact is very close to the task of natural language inference (nli). nli is a sequence classification task which takes two inputs-a hypothesis and a premiseand produces one of the possible outputs: the hypothesis is entailed by (follows from) the premise, contradicts the premise, or their relation is neutral. recently, neural models for nli @xcite reached near-human levels of performance and nli was used for evaluating the output of abstractive summarization systems @xcite . this brings a question: can we use an nli model for evaluating the semantic accuracy of d2t outputs? the main idea of our method is to check with a general pretrained nli model if the semantic information implied by the input data and the generated text is equal. we achieve this by using the nli model to check for entailment in two directions: by inferring input facts from the generated text we can check for omissions, while the other direction allows us to check for hallucinations. 1 for instance, consider the two input facts from figure 1 applying nli for the d2t task brings a problem: the hypothesis for the standard nli task is a natural language text, but the input for d2t generation is structured. however, we show that we can easily sidestep this issue by transforming the data into text using a trivial template for each fact. we demonstrate that even without any human references or in-domain training and with minimal handcrafting, our approach achieves high accuracy (>90%) on the e2e challenge data @xcite , competitive with scripts specifically handcrafted for the domain, and produces useful results (>75% accuracy) on the more challenging webnlg dataset @xcite . a manual error analysis shows that some instances marked as errors were in fact assessed correctly by our metric; we also identified a few major sources of errors that can be mitigated by in-domain tuning. the experimental code for our metric is now available on github. foot_2 2 related work automatic evaluation of nlg nlg outputs were traditionally evaluated by reference-based metrics measuring n-gram overlap with a reference, such as bleu @xcite , rouge @xcite and meteor @xcite . alternative, referenceless quality estimation metrics based on language model scores @xcite or linguistic features @xcite focus on fluency and do not consider semantic accuracy. recent works try to estimate nlg output quality with finetuned pretrained models @xcite @xcite . the score from these models can capture some aspects of semantic accuracy, but only implicitly. ## experimental setup we experiment with two recent english data-totext datasets with a triple-like format: webnlg @xcite and e2e @xcite . 4 since both of them were used in shared tasks, sets of system outputs and measures of semantic accuracy are available (see supplementary for details). for webnlg, we compare our metric with crowdsourced human ratings of semantic adequacy @xcite . human annotators used a three-point likert scale (1 = incorrect, 2 = medium, 3 = correct) and answers are averaged over multiple annotators. in our experiments discussed in section 5.1, we consider a sentence correct if it achieved human rating 2.5 or higher (we also tried a threshold of 2.0, with slightly worse results). for the e2e dataset, the challenge results were checked for semantic accuracy using a handcrafted automatic script @xcite ), foot_5 we therefore use this automatic script as the ground truth for evaluating our metric in section 5.2. we further use small sets of system outputs and humanwritten texts with expert annotation (provided by @xcite to evaluate our approach against gold-standard annotation and to compare to existing semantic accuracy classifiers for e2e data in section 5.3. we evaluate the default and backoff approaches to acquiring templates as described in section 3.2. the default setup works with one custom template per predicate type. for webnlg, we obtained templates by delexicalizing human references for single-triple examples from webnlg training data. 6 for e2e, we handcrafted 8 templates. the templates are filled with values from individual input triples and concatenated for multitriple inputs as described in section 3.3. ## results analysis we evaluate our metric in terms of accuracy, precision, recall, and f1-measure (where not_ok samples are treated as positive since we focus on detecting errors). we additionally perform a manual error analysis on a random sample of 100 error examples for each dataset, i.e. examples where our metric gave a different assessment from the ground truth (provided by crowdsourced annotation for webnlg and by a handcrafted classification script for e2e as described in section 4). in general, the results are high above the random baseline (0.5 for the rough metric and 0.25 for the fine metric) but differ between the datasets, which we discuss below. ## conclusions and future work we described an automatic metric for evaluating semantic accuracy of d2t generation. with just a basic setup, without human references or training and with minimum handcrafting, our metric is able to detect omissions or hallucinations in generated texts, with results competitive with crowdsourced human ratings or handcrafted scripts customized for particular domains. while our metric seems to scale well to more complex inputs in our experiments on the webnlg and e2e data, we note that these datasets are still relatively limited. further experiments are needed to evaluate this approach on long text generation and tasks where content selection is required, which we reserve for future work. we also plan to integrate our metric as a reranker into an nlg system and apply small-scale in-domain finetuning in order to further improve results. following our findings from the error analysis on webnlg, which showed that human ratings of semantic correctness are influenced by grammaticality, we would like to investigate the possibilities for combining our metric with a fluency/grammaticality checker @xcite , as well as ways to better separate these two criteria in human evaluation.