Assessing Semantic Consistency in D ata‐to‐ T ext Generation: A Meta-Evaluation of Textual, Semantic and Model-Based Metrics (2025)

Abstract: Ensuring semantic consistency between semantic-triple inputs and generated text is crucial in data‐to‐text generation, but continues to pose challenges both during generation and in evaluation. In order to assess how accurately semantic consistency can currently be assessed, we meta-evaluate 29 different evaluation methods in terms of their ability to predict human semantic-consistency ratings. The evaluation methods include embeddings‐based, overlap‐based, and edit‐distance metrics, as well as learned regressors and a prompted ‘LLM‐as‐judge’ protocol. We meta-evaluate on two datasets: the WebNLG 2017 human evaluation dataset, and a newly created WebNLG-style dataset that none of the methods can have seen during training. We find that none of the traditional textual similarity metrics or the pre-Transformer model-based metrics are suitable for the task of semantic consistency assessment. LLM-based methods perform well on the whole, but best correlations with human judgments still lag behind those seen in other text generation tasks.

## introduction the last few years have seen substantial advances in the quality of automatically generated text @xcite in particular with respect to suprasentential fluency and coherence, thanks to pretrained language models with ever larger numbers of parameters, trained on ever larger datasets @xcite . however, these advances have come at the price of semantic controllability, with confabulation, replacement and omission of content all commonly found in state-ofthe-art text generation systems @xcite . in controlled text generation, whether in the form of data-to-text generation, or of free text generation with given control attributes, some or all of the input must be matched in specific ways by the output. ensuring this semantic consistency between input and output is a known weakness of otherwise stateof-the-art neural generators, and we currently do not have sufficiently reliable methods either (i) for ensuring semantic consistency as part of the neural generation process, or (ii) for evaluating systems in terms of the degree to which they achieve it. this poses particular problems for tasks like data-to-text generation @xcite , where the information conveyed by the output is intended to be entirely controlled by the information contained in the input. currently the only way to perform such semantic consistency assessment reliably is manual evaluation, with supervision-trained automatic methods getting up to about three quarters of unseen assessments right @xcite @xcite . an automatic method that can reliably assess the semantic consistency between inputs and outputs directly would be useful not only for evaluation in and post development, but also potentially as part of the text generation method itself, e.g. via reranking of outputs @xcite , or as part of a loss function. however, semantic consistency continues to be challenging to assess automatically with a high degree of reliability, and evaluation by comparison to human-written reference outputs or by human evaluators is predominantly used instead. in this paper, we evaluate diverse types of semantic consistency evaluation methods on two datasets: the webnlg 2017 human evaluation dataset (dataset a), and a newly created, unseen webnlg-style dataset (dataset b). moreover, we test each method under two conditions: (i) with triple inputs 'textified' before assessment, and (ii) without. our main contributions are: ## related work semantic consistency assessment (sca) is considered challenging @xcite @xcite . existing methods tend to not directly use the structured meaning representations (smrs) that typically form the input to data-to-text generation, but first (trivially) map them to text, before applying either a semantic similarity measure @xcite , or performing a natural language inference (nli) task @xcite , on the (mapped) inputs and corresponding textual outputs. such methods tend to not generalise well beyond the data they were trained on. moreover, pipelining multiple processes aggregates errors. @xcite presented an end-to-end data-to-text generation system with a semantic fidelity classifier for semantically inaccurate text detection. @xcite proposed an automatic metric for assessing entity-based semantic adequacy of rdf verbalisers. ribeiro et al. (2020) used natural language inference (nli) to detect two-way entailment between generated text and the input. in existing work, semantic consistency assessment has been construed e.g. as binary classification @xcite , mutual textual entailment @xcite , or thresholded semantic similarity @xcite , between inputs and outputs (see also related research in sec-tion 2). however, little attention has been paid to underlying definitions of semantic consistency and the role they play in evaluation results which can vary substantially depending on which definition underlies an evaluation method. ## datasets we evaluate two corpora of rdf-text pairs with human judgments of semantic adequacy: dataset a (webnlg 2017): a subset of the public webnlg 2017 human evaluation data @xcite , comprising 100 randomly sampled input-output pairs (out of 2,230 total) across ten systems. each text was rated by three annotators on a 3-point scale for semantic adequacy ("does the text correctly represent the input triples?"), and we use the mean score. ## evaluation methods we meta-evaluate 29 evaluation methods in terms of their ability to predict human semantic consistency ratings; the methods fall into three broad categories depending on the underlying (implied) definition of semantic consistency: we deliberately include both long-established metrics and more recent model-based methods. while some traditional overlap or distance-based metrics are considered outdated in text-to-text evaluation, they continue to be used in some parts of the research community and therefore remain relevant for comparison. their inclusion also allows us to quantify more precisely how much newer metrics and llm-based evaluators improve upon them in the more challenging data-to-text setting, thereby providing a comprehensive baseline for future work. more specifically, we use the following 29 automatic methods (see appendix c for details) falling into five method types: bertscore p/r/f 1 ; sbert cosine, dot, euclidean, manhattan. the above methods are used to obtain semanticconsistency assessments on data-to-text input/output pairs in our two datasets, with and without textification of inputs. ## experimental set up our aim is to determine which of the metrics from the preceding section best predict human judgements of semantic consistency. in data-to-text generation, human evaluation remains the gold standard, but is expensive and time-consuming @xcite . consequently, automatic proxies are widely used, including embeddings-based (bertscore @xcite ; sbert @xcite ; bleurt (sellam et al., 2020)), overlap (rouge @xcite ; sacrebleu @xcite ; meteor @xcite ), and edit/set measures (levenshtein @xcite ; jaccard/dice @xcite ). we assess all of these in our experiments. however, they were designed for textto-text tasks, not structured inputs, so we test them both with and without textification. we extend our tests to llm-as-judge methods, prompting three pre-trained models (command-r-plus, llama3-70b, mistral-7b) to rate semantic consistency on a 1-3 scale, and averaging over three seeds (42; 1234; 1738). we evaluate the metrics and llm judges on the two datasets from section 3, once with inputs as they are (structured triples), and once with linearised and 'textified' triples (see algorithm 3 in appendix a). our simple textification method replaces special characters with spaces, collapses whitespace, and trims edges. we compute scores with all evaluation methods, then compute pearson's r between the scores and mean human ratings for each dataset, ranking methods by correlation to identify the strongest predictors. ## result and analysis the results in first, textification exerts a positive effect on most of the evaluated metrics on dataset a (except bleurt, rouge-l recall, rouge-1 recall, sacrebleu, jaccard distance), yielding an average pearson's r increase of about 0.07; and also on dataset b (here the exceptions are the four sbert metrics, rouge-2 precision, sacrebleu, levenshtein distance, jaccard distance) where the average increase in r is about 0.04. textification has a slightly smaller beneficial effect on dataset b correlations for the llm judges, except mistral. in stark contrast, it causes llm correlations to collapse across the board on dataset a (see also discussion section). among the metrics, the four sbert-based similarity metrics with textification attain the four highest correlations on dataset a (@xmath0.582), substantially outperforming bleurt (0.530). in contrast, bleurt (@xmath1.733), followed by bertscore f1 (0.670) and then the sbert variants (0.655) on dataset a, the llm judges without textification, and the model-based metrics with textification, perform best overall and more or less on a par. on dataset b, the standout performances are by bleurt, the llm ensemble and command-rplus by a considerable margin, and these are hardly affected by textification. moreover, on dataset b, the more traditional metrics fail almost entirely (see rows from infersent down to jaccard). ## discussion among the clearest patterns in our results is the uniformly poor performance of traditional textual similarity metrics at the semantic consistency assessment task. infersent and universal sentence encoder, the two neural but pre-transformer metrics, can also be dismissed for this task. another clear pattern is that while the traditional textual similarity metrics perform very similarly on dataset a and dataset b, every evaluation method that involves an llm sees a jump in correlation from dataset a to b. it's not entirely clear why the latter should agree more with humans on b than on a. it may in part be due to the fact that outputs in dataset b are generally of higher quality, perhaps displaying more minor as well as fewer errors. textification caused the llm-judge scores to collapse (while improving almost all other scores) on dataset a. the latter was almost certainly ingested as part of their training data by the four llms tested, so textification had the net effect of obscuring whether an input and output belonged together (hence their semantic relatedness). it is worth noting that the highest correlations with human judgments in our experiments were just over @xmath2.7. while such values fall below textto-text generation tasks where correlations above 0.8 are commonly reported, they nevertheless represent the current performance ceiling for semantic consistency in data-to-text generation. this gap highlights both the greater intrinsic difficulty of the task and the lack of specialised metrics designed for structured input-output matching. the poor performance of older metrics confirms their unsuitability for semantic consistency assessment, and provides a baseline against which to measure progress with newer methods. ## conclusion we have presented a meta-evaluation of 29 different evaluation methods on the task of assessing semantic consistency between triple inputs and corresponding textual output in data-to-text generation. the evaluation methods came in three broad flavours (reflecting the underlying definition of semantic consistency): textual similarity, semantic similarity, and model assessment. metaevaluation on the webnlg 2017 human evaluation dataset and a newly created webnlg-like dataset revealed that none of the older metrics, including those based on pre-transformer neural models, are suitable in the least for the data-to-text semantic consistency assessment task. at the same time, no method surpassed pearson's @xmath3.635 (llm ensemble) on dataset a, or @xmath4.733 (bleurt) on dataset b, with the best prompted llm-as-judge evaluation methods close behind. these results underscore the notable gap in correlations with human assessments between data-to-text and text-to-text evaluation benchmarks; in the latter, correlations well over 0.8 are routinely reported. one route that may be worth exploring in future work is different ensembling strategies combining the best embeddings-based metrics with llm-as-judge methods. improvements in sca methods can in turn lead to improvements in both data-to-text evaluation and generation. finally, by evaluating both older and state-ofthe-art methods side by side, we have provided a comprehensive picture of the current landscape. while some of the older metrics are known to be less suited for the purpose of sca, their continued use in the community and their role as comparative baselines make their inclusion informative. our experiments were restricted to webnlg-style english datasets, and it will be important to test whether the observed patterns hold across different domains and languages, to establish the generalisability of the findings.