Trainable Ranking Models to Evaluate the Semantic Accuracy of Data-to-Text Neural Generator (2021)

Abstract: In this paper, we introduce a new embedding-based metric relying on trainable ranking models to evaluate the semantic accuracy of neural data-to-text generators. This metric is especially well suited to semantically and factually assess the performance of a text generator when tables can be associated with multiple references and table values contain textual utterances. We first present how one can implement and further specialize the metric by training the underlying ranking models on a legal Data-to-Text dataset. We show how it may provide a more robust evaluation than other evaluation schemes in challenging settings using a dataset comprising paraphrases between the table values and their respective references. Finally, we evaluate its generalization capabilities on a well-known dataset, WebNLG, by comparing it with human evaluation and a recently introduced metric based on natural language inference. We then illustrate how it naturally characterizes, both quantitatively and qualitatively, omissions and hallucinations.

## introduction data-to-text (d2t) generation @xcite @xcite is a specialized task of natural language generation (nlg) where a model takes as input (semi)-structured data (e.g. a table) and generates a textual utterance that is both syntactical and semantically faithful to the input. several architectures were proposed to solve this task. they may rely strictly on templates @xcite @xcite , separate planning (what to say) from generation (how to say it) @xcite or be a fully derivable neural architecture @xcite @xcite . while achieving interesting performance at natural language generation tasks @xcite , pre-trained neural language models (hence neural architectures in general), are prone to hallucinate facts @xcite which brings their usability at stake in sensitive domains such as the legal one. in this paper, we wish to promote the usability of neural architecture by proposing a new trainable automatic evaluation metric well suited to evaluate the semantic accuracy of such d2t generator. this metric is designed as a two factor "round-trip evaluation" in order to assess the accuracy a given generated hypothesis. first, we use the hypothesis to try to recreate the original table by ranking its values amongst all other values in the dataset. then, we retrieve similar references amongst all other references in the dataset by ranking them still using that same hypothesis. we illustrate both round-trip evaluation scheme (table reconstruction and reference ranking) in figure 1 . our approach is well suited to semantically and factually assess a generator's performance in cases where the tables can be associated with multiple references, and the tables' values contain textual utterances. we present how one can further specialize the proposed evaluation metric by training the underlying ranking models on the target dataset, hence providing a more robust evaluation. re-lying on the mean average precision, we present how it naturally characterizes, both quantitatively and qualitatively, omissions and hallucinations of a given generator. this framework offers great flexibility in how it can be implemented and further improved. we identify two main components that can be tuned to improve the efficiency of our proposed metric when evaluating nlg systems; a similarity function between reference texts and the underlying ranking models. however, having a metric where the efficiency is highly dependent on how it is implemented is highly problematic for an absolute comparison against other metrics or evaluation methodologies. this is why we propose a way to "fix a metric" i.e. how good, on the gold annotations, one implementation of the metric can be? having a "fixed metric" then allows us to evaluate nlg systems against each other properly. in our experiments, we first apply our metric on a challenging dataset in the legal domain, plum2text @xcite . we show how the specialization of the ranking models can be beneficial or even necessary. more precisely, we illustrate these benefits when paraphrasing between the data (i.e. table) and the reference text highly characterizes the dataset in hand. then, we illustrate its generalization capabilities even in simpler settings on a well-known d2t dataset, webnlg @xcite . we show how it is able to discriminate a set of generators, and correlates positively with human judgment. our contribution is thus a new trainable automatic d2t evaluation metric that naturally characterizes both omissions and hallucinations of neural architectures. ## evaluating data-to-text generation evaluating natural language generated text is a very hard task. @xcite and @xcite question the validity of widely used metrics. sai et al. (2020) provide an extensive survey of the field, and more precisely separate the d2t evaluation metrics along 2 dimensions: either they use the table t or not (i.e. table-free), and either they are trained or untrained metrics. for instance, automatic evaluation metrics such as bleu @xcite , rouge @xcite or meteor @xcite are not trained and neither use the table t. they only partially account for the faithfulness of a given generated hypothesis w.r.t its associated references. even though these metrics are widely used, they fall short to capture factual aspects in a d2t setting, and correlate poorly with human judgment @xcite . bleurt @xcite ) is a metric trained on english texts that is designed to better model human judgment on generated texts. however, it does not take into account the input table. @xcite proposed ibleu for paraphrase generation, an adaptation of the bleu score that takes into account the context (the original phrase), the generated hypothesis and the reference. its variant, bleu-t, rewards an hypothesis h that overlaps with the content of the input table t as follows; bleu-@xmath0, t) + (1 -α)bleu(h, r) where α is a parameter that balances faithfulness between the table t and the reference r. table-free table untrained bleu, rouge, meteor parent, bleu-t/ibleu, nli, ours trained bleurt ours trained metrics using the context (in our case t) have been proposed for dialogue generation tasks @xcite . however none of them is suitable for a d2t setting where the characterization of hallucinations (and to some extent omissions) are required in order to use a neural d2t generator in production. we thus wish to fill in this gap by presenting in the following section a new evaluation scheme. this scheme offers the advantage to exploit both the table t and the reference r, to be based on ranking models, and to be either trained or not. we illustrate in ## data-to-text evaluation through ranking to assess the accuracy of a generated hypothesis h, it can be useful to consider both the input table t and the target reference r, i.e. validate the correctness of h according to its table and its reference. we thus propose a way to assess the fidelity of h by reconstructing the table t (h ) t) and by retrieving its corresponding reference(s) r (h ) r) using ranking models. this premise is highly motivated by the fact that similar text descriptions should be associated to semantically similar table contents. in both settings, the ranking models are evaluated using the average precision (ap), which we describe next. ## experiments in this section, we first illustrate the benefits of fine-tuning our proposed metric on the plum2text target dataset @xcite . we then show how our metric can discriminate generators, and analyze omission and hallucination rates using webnlg @xcite . ## conclusion in this paper, we introduced a new trainable automation evaluation metric relying on ranking models which is specific to the d2t setting. to the best of our knowledge, this is the first metric that naturally quantifies omissions and hallucinations of neural textual generators that can also handle paraphrases. characterizing omissions and hallucinations are of important matter in a sensitive area such as the legal domain. this lack of characterization is often a blocker to the use of recent neural generator models in the legal field. we hope that this metric will also promote the use of recent advances in neural textual generation in sensitive domains such as the medical field. in our future works, we would like to use our metric to guide the decoding steps of neural d2t generators in order to produce faithful textual descriptions. as previously mentioned, the metric that we proposed is well suited to semantically and factually assess a generator's performance in cases where the tables can be associated with multiple references, and the tables' values contain textual utterances. we wish to generalize the way our method ranks table values through relevance matching @xcite . this would be highly desirable in cases where table values are only up to a few tokens.