Data- Q uest E val: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021)

Abstract: QuestEval is a reference-less metric used in text-to-text tasks, that compares the generated summaries directly to the source text, by automatically asking and answering questions. Its adaptation to Data-to-Text tasks is not straightforward, as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting metric is reference-less and multimodal; it obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. We make data-QuestEval’s code and models available for reproducibility purpose, as part of the QuestEval project.

## introduction data-to-text generation (dtg) aims at generating descriptions in natural language given a structured input, e.g. a table @xcite . reliability and precision of generated texts is currently regarded as a major issue in dtg @xcite , with experimental surveys showing that real-life end users of dtg systems care more about accuracy than about readability @xcite . neural nlg systems are known to be fluent, but prone to hallucinations @xcite , i.e. they tend to include nonfactual information. however, their evaluation remains an open research problem @xcite . a recent approach, questeval @xcite has shown significant improvement over standard metrics on summarization tasks. to measure semantic matching between an evaluated summary and its source document, questeval relies on question generation and answering (qg/qa) systems. precision q g q a questions (q y ) scores ( eq.1 ) table table description as illustrated in figure 1 , a qg system generates a set of relevant questions conditioned on the source document, which are then asked on its generated summary. conversely, questions generated from the summary are answered using only the source input. if the answers provided by the qa systems are correct, the summary is deemed consistent with its source document. can questeval be adapted for evaluation on dtg tasks? so far, questeval's qg/qa systems have been trained on a purely textual dataset, squad @xcite , which restricts the evaluation to comparisons between two texts. unfortunately, dtg inputs are of different modalities than text (e.g. structured tables). in the absence of specific multimodal-qa datasets, how can one obtain these multimodal-qg/qa models required for a data-questeval? to fill this gap, we propose an effective method for creating synthetic multimodal-qg/qa datasets, by relying only on existing, purely textual, qg/qa datasets. trained on such synthetic multimodal datasets, qa and qg models can now be used in questeval, enabling direct comparison between an evaluated text and its structured input, removing the need for costly gold references. furthermore, this method does not rely on any task-specific annotated qa dataset, which makes the approach general and suitable for any dtg task. ## related work based on n-grams similarity between an evaluated text and its gold references, bleu @xcite and parent @xcite are the two standard metrics reported in dtg evaluations. beyond n-grams, @xcite proposed to use a model trained on the reverse task, i.e. text to data reconstruction, and to compare the two data generated i) from the reference, and ii) from the hypothesis. @xcite introduced bertscore, where texts are compared given the contextual bert representations of their tokens. however, all these metrics require gold references. in a first attempt for a reference-less evaluation metric in dtg, @xcite proposed to train a neural model to directly predict the human ratings; this requires a significant amount of human annotated data, and is eventually biased towards the annotator rather than the task @xcite . concurrently, a family of reference-less metrics has emerged in summarization @xcite . the amount of information shared between two texts is measured by generating questions on the source text, and asking them on the evaluated text. in its recent extension, questeval @xcite was shown to overperform standard metrics in text-vs-text tasks such as the evaluation of summaries. unfortunately, text-qg/qa systems are not usable off the shelf for tasks in other modalities which require multimodal-qg/qa systems. further, there is significant variability in structures (e.g. tables, knowledge graphs, etc.) and domains (e.g. biographies, sports, etc.) across dtg tasks @xcite . therefore, generalizing questeval to dtg tasks relying solely on existing multimodal-qg/qa datasets is not a promising direction: the effectiveness of data-qg/qa models would be limited to the specific structures of the few existing data-qa datasets -e.g. the wikitable-questions benchmark @xcite ; moreover, it is unrealistic to annotate qg/qa corpora for each specific modality/domain. conversely, our proposed method generalises to any structured data, given only a text-qa dataset. data-to-text corpus references ta ble s synthetic data-qa corpus questions ta ble s text-qa/qg corpus (e.g squad) text-qg model data-qg data-qa & answers 1 2 3 4 figure 2: synthetic corpus creation we are able to create a dataset of (table, question, answer) triples, by transcribing references into questions via a textual qgmodel trained on squad. numerals refer to steps explained in section 3.2. ## experiments in this paper, we aim to evaluate the effectiveness of our proposed multimodal adaptation of questeval. consistently with previous works @xcite , metric performance is measured by how much it reflects human judgement, assessed via pearson correlation coefficient. ## conclusion in this work, we proposed an efficient method to evaluate in a reference-less multimodal setup of data-to-text generation. to train data-questeval on different modalities, we propose a methodology to create synthetic corpora of (data, question, answer) triples from any data-to-text generation task. we show that our approach outperforms several existing metrics w.r.t. human judgement on two standard data-to-text generation benchmarks, and performs reasonably well when trained on a different domain. while qg/qa modeling per se is out of the scope of this paper, further improvements could be obtained thanks to advances in multimodal qa, a field particularly prolific @xcite @xcite . in future works, we plan to study how qg/qa systems perform on more complex and abstractive datasets.