Semantic Evaluation of Multilingual Data-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores (2025)

Abstract: Performance in the KG-to-Text task has improved over the years, particularly in English. However, models are still prone to mistakes like Additions and Omissions. Furthermore, few languages are taken into account since both train and test data are not readily available. In this paper, we hope to facilitate the development and improvement of multilingual KG-to-Text models by providing a multilingual evaluation framework that is reference-less (no need for test data) and permits estimating how much a KG-to-Text Model under- (omission) or over- (addition) generates. We focus on two high (English, Russian) and five low (Breton, Irish, Maltese, Welsh, Xhosa) resource languages and show that our metric has fair to moderate correlation with reference-based metrics, positioning it as a consistent alternative when no references are available. We also show that our metric outperforms prior reference-less metrics in correlation with existing human judgments. Additional human evaluation shows moderate to strong correlation with human annotators in assessing precision and recall at a higher granularity level than shown in previous studies. Since our metric provides scores for precision and recall, it helps better assess the level of over- or under-generation of multilingual KG-to-Text models.

## related work zero-shot nli classification for semantic accuracy. @xcite proposed to evaluate the semantic accuracy of english kg-to-text generation by leveraging the zero-shot abilities of the english-based roberta-large-nli model @xcite 2 . in their work, they try two approaches: one to search for omissions and one to search for additions. in the first one, they use the entire generated text as a premise and iterate over every individual fact from the input graph 2 https://huggingface.co/facebookai/roberta-lar ge-mnli as a hypothesis, marking facts not classified as entailments as omissions. in the second approach, they use the entire input graph as a premise and the generated text as a hypothesis, marking the presence of additions when the text is not classified as entailment. this approach is tested exclusively in english and, while it provides high granularity when measuring omissions, it is less specific when measuring additions. fine-tuning on synthetic data for factual faithfulness. @xcite went a step further and fine-tuned an english model, first an electra-base-discriminator @xcite foot_1 then a deberta-v3-base (he et al., 2021) 4 , on synthetic data to detect whether a given fact is present in a generated text (akin to @xcite . their training data consist of real positive (text, fact) pairs and synthetic negative (text, fact) pairs made of 90% type i errors (where they perturbed the fact by changing its subject, predicate, and/or object) and 10% type ii errors (where they perturbed the text by removing one or both entities from the fact and/or the n-grams most similar to the predicate). like factspotter, this approach was tested exclusively in english and, while it has a high granularity when measuring omissions, it does not directly address additions. beyond kg-to-text: bidirectional entailment has been tried as a way of evaluating summarization. @xcite and @xcite proposed reference-based approaches, while chen and eger (2023) proposed a reference-less method. ## method to learn our metric, we fine-tune an existing multilingual natural language inference (nli) model by adjusting its classification head to perform regression instead. we then train on data created to capture different combinations of precision and recall using binary cross entropy (bce) loss. given a premise and a hypothesis, nli models predict if the hypothesis is entailed, neutral, or contradicted by the premise. for precision, we check if the text is entailed by the graph (how much of the text can be inferred from the graph). for recall we check if the graph is entailed by the text (how much of the graph content can be inferred from the text). we are not interested in the 3 classes from the nli head, only in the strength of the entailment between premise and hypothesis. we fine-tune the nli classifier as a regression model by focusing only on the entailment weights from the classification head instead of the three existing output classes. we train simultaneously for precision and recall by swapping the graph and text order and targeting the respective score. the f1 score is computed as usual (equation 1 ) by taking the harmonic mean of precision (p) and recall (r). this score functions as a high-level proxy for semantic faithfulness: the higher the f1 score, the higher the semantic similarity between the graph and the text. ## conclusion previous work on reference less evaluation of kgto-text generation has mainly focused on english, providing global metrics for semantic faithfulness. we extend this work by presenting models which support the reference less evaluation of multilingual kg-to-text generation while allowing for a finer-grained evaluation in terms of precision, recall and f1. the proposed models show strong correlation with human judgments of precision and recall for several languages, moderate to strong correlation with automatic metrics and high retrieval accuracy. on a small data set of 10k (graph,text) pairs retrieval accuracy is high indicating that the representations learned by our model provide a good basis for identifying matching kg/text pairs. testbed, supported by a scientific interest group hosted by inria and including cnrs, renater and several universities as well as other organizations (see https://www.grid5000.fr ). this work was also granted access to the hpc resources of idris under the allocation ad0110165612 made by genci.