Incorporating Formulaicness in the Automatic Evaluation of Naturalness: A Case Study in Logic-to-Text Generation (2025)

Abstract: Data-to-text natural language generation (NLG) models may produce outputs that closely mirror the structure of their input. We introduce formulaicness as a measure of the output-to-input structural resemblance, proposing it as an enhancement for reference-less naturalness evaluation. Focusing on logic-to-text generation, we construct a dataset and train a regressor to predict formulaicness scores. We collect human judgments on naturalness and examine how incorporating formulaicness into existing metrics affects alignment with these judgments.

## introduction when generative models are provided with structured input data (e.g., logical formulae, tables, rdf triples, etc.) and are tasked with generating textual descriptions of this input data, they sometimes produce outputs that mirror the structure of the input very closely. this can be undesirable in domains and applications that require fluent output, instead of the stilted texts exemplified in these considerations have implications for evaluating the broader notion of naturalness in text generation. one important but previously unexplored feature is what we term formulaicness: the degree to which a data-to-text nlg system's output explicitly preserves the structural form of its input. high formulaicness reflects a close, template-like correspondence to the input, while low formulaicness indicates greater abstraction or paraphrasing. as shown in in this paper, we propose a methodology for measuring formulaicness and investigate whether incorporating this measure into other referenceless @xcite evaluation metrics (including llm-based ones) can improve the automatic assessment of naturalness. such metrics typically capture features like fluency, grammaticality, or readability (e.g., @xcite @xcite @xcite , but are not always consistent with human judgments @xcite . our main question is: does incorporating formulaicness in evaluation metrics improve the automatic assessment of naturalness? we focus on logic-to-text generation, a task with a long tradition in nlg (e.g., @xcite that has drawn significant renewed interest more recently (e.g., @xcite , in part due to the growing attention to the reasoning abilities of llms @xcite , with benchmarks and experimental work carried out, also regarding first-order logic (e.g., @xcite @xcite . logic-totext generation can be particularly valuable when logical formulae are difficult to interpret, for instance for beginners @xcite , since translating them into natural language can enhance comprehension. for example, @xcite employed a logic-to-text system to translate first-order logic formulae into english in a classroom setting. ## methods we constructed a dataset consisting of texts (a.) derived from logical formulae paired with formulaicness scores in the [0, 1] range (b.). we used this subtask dataset input output model logic-to-text grade grinder corpus (barker-plummer et al., 2011) there exists a cube x that is large, and for all y and z, if y is a cube and large and z is a dodecahedron, then x is y and z is not behind y. the data provided shows the results of different genres of text, including news with and without a star, wiki, and web, in terms of the number of sentences, length, yield, and precision. the news with a star has one hundred sentences, an average length of nineteen point three words, a yield of one hundred forty two, and a precision of seventy eight point nine percent. llama-3.3-70b-instruct @xcite rdf-to-text webnlg @xcite [ data to train a regressor to predict formulaicness f (t) for a given text t (c.). we collected human judgments of naturalness (d.) and explored the impact of incorporating formulaicness into other metrics (m (t)) (e.) using a weighted average: we used the complement 1 -f (t), so as to minimize formulaicness, e.g., in contexts where greater abstraction from the input is desirable. the weights α and β control the relative influence of m and f . a. texts we used a subset of the dataset from calò et al. (2022), consisting of first-order logic formulae from the grade grinder corpus (ggc; barker-plummer et al. 2011), paired with english translations generated using three systems, which generate texts of different degrees of formulaicness by design: (i) a system that generates literal translations of the formulae; (ii) ranta (ranta, 2011b), which performs syntactic optimizations to improve fluency; and (iii) lola, an extension of ranta that applies logical optimizations to the input formula before verbalizing it (appendix a for details). to complement the set, we included (iv) the humanwritten translations of the formulae; henceforth, we speak of literal, ranta, lola, and human. b. scores in calò et al. (2022), the three systems were ranked @xmath0, based on human judgments, using trueskill @xcite , an algorithm for estimating relative performance scores, which returns a mean (µ) and a standard deviation (θ) for each system reflecting the final ranking (the higher the µ, the better the system). we heuristically interpret this fluency ranking as a proxy for formulaicness, with human-written texts considered the least formulaic overall. to assign appropriate formulaicness scores to texts (i.e., ensuring @xmath1, we defined score bins for each system by leveraging the µ and θ values returned by trueskill. 1 the center of each bin was set to the corresponding trueskill µ (with higher µ indicating lower formulaicness; values were normalized to the [0, 1] range), and the bin width was set @xmath2, 1]). within each bin, we randomly sampled floatingpoint values and took their complements (1-x) to synthesize a formulaicness score for each text. the final dataset consists of 570 texts and corresponding formulaicness scores. we split the data into training, validation, and test sets using a 70-15-15 ratio, stratifying by formulaicness score and text length by token to ensure consistent distributions across splits. inter-annotator agreement (iaa) was computed using krippendorff's α @xcite and ranged from low to moderate across the four batches (0.19, 0.22, 0.43, 0.13), with an average of 0.25. variation across batches can result from differences in item difficulty and annotator behavior, including variation in likert-scale leniency. these iaa scores are consistent with prior findings that human judgment tasks in nlg often yield low agreement (van der @xcite . importantly, low iaa is not inherently problematic. it may reflect the subjective and challenging nature of the task, which is itself informative. for instance, @xcite argues that human label variation is ubiquitous and that high iaa is typically achieved only under artificial conditions. perplexity @xcite ; the syntactic log-odds ratio (slor, @xcite ; llama 3.1 8b instruct as llmas-judge (llama, @xcite . to make all metrics interpretable in a consistent way, where higher is better (= more natural), we inverted ppl and slor (henceforth, ippl and islor). ## results for each text in the test set, we computed formulaicness, gruen, fre, ippl, islor, llama, and mean naturalness scores provided by participants. all baseline scores were normalized to the [0, 1] interval. for each baseline metric, we also computed combined score (equation 1 ). additional information is provided in appendix f. the pearson correlation between predicted formulaicness scores and average human ratings is @xmath3.594 (p ≪ 0.005), indicating a highly significant negative correlation: more formulaic texts tend to be rated as less natural. next, we evaluated the five baseline metrics, both in isolation and combined ( to assess the contribution of formulaicness when combined with baseline metrics, we conducted goodness-of-fit tests by fitting linear regression models to predict human naturalness ratings using: (i) each baseline metric alone, (ii) formulaicness alone, and (iii) their combinations ( there is an element x such that there is an element y such that there is an element z such that x is a cube, x is small, y is a cube, y is medium, z is a cube and z is large. some dodecahedron is not small. 4.40 0.040 medium human naturalness (somewhat awkward phrasing); very low formulaicness, likely due to lack of literals or formulaic patterns. to investigate discrepancies between human judgments of naturalness and formulaicness scores assigned by the bert regressor, we performed a linguistic analysis on samples where human naturalness ratings diverged significantly from the bert regressor predictions (see to examine how adding formulaicness affects baseline metrics, we selected examples with the largest deltas between baseline scores and scores combined with formulaicness ( ## discussion and conclusion formulaicness improves the evaluation of naturalness. returning to our research question, the results in §3 suggest that formulaicness offers an effective enhancement to the automatic evaluation of naturalness in logic-to-text generation. it aligns well with human judgments and consistently improves the performance of baseline metrics in terms of correlation with human ratings (