Enhancing and Evaluating the Grammatical Framework Approach to Logic-to-Text Generation (2022)

Abstract: Logic-to-text generation is an important yet underrepresented area of natural language generation (NLG). In particular, most previous works on this topic lack sound evaluation. We address this limitation by building and evaluating a system that generates high-quality English text given a first-order logic (FOL) formula as input. We start by analyzing the performance of Ranta (2011)’s system. Based on this analysis, we develop an extended version of the system, which we name LoLa, that performs formula simplification based on logical equivalences and syntactic transformations. We carry out an extensive evaluation of LoLa using standard automatic metrics and human evaluation. We compare the results against a baseline and Ranta (2011)’s system. The results show that LoLa outperforms the other two systems in most aspects.

## introduction logical formalisms play a pivotal role in many areas of science. hence, grasping the meaning of these formalisms is crucial for many scholars and researchers. however, this task is not straightforward, and sometimes even experienced logicians might have trouble deciphering a complex formula. natural language generation (nlg) techniques can be employed to ease this task. however, logicto-text generation is understudied, compared to text generation from other inputs @xcite . one notable exception (see §2 for some other examples) is @xcite , a rule-based system that translates between first-order logic (fol) formulae and natural language (nl). while providing a promising starting * these authors contributed equally to this work. point for logic-to-text generation, the system is not evaluated. in our work, we first address this gap via a human translation quality assessment (tqa). based on this, we propose lola, a novel logic-totext system extending @xcite 's architecture, which searches for the most suitable formula for translation among the pool of logically equivalent formulae. we also address one of the many issues that make nlg evaluation challenging @xcite , namely, defining the core dimensions to evaluate @xcite , especially issues of meaning vs. grammaticality. these issues come to the fore in logic-to-text generation, where text should be faithful to the original formula, comprehensible, and fluent. these are the central requirements to look for, as text generated from logic can be extremely disfluent and incomprehensible (e.g., a literal translation from a formula), while still being faithful. furthermore, evaluating faithfulness cannot rely on checking factual accuracy (as in, e.g., webnlg @xcite ), due to the problem of logical form equivalence @xcite , which implies that every formula of fol is equivalent with infinitely many other fol formulae, where the question of whether two fol formulae are logically equivalent is, in general, undecidable. this complicates the problem of finding a formula that is most suitable for being input to an nlg program. there are also potential trade-offs between evaluation dimensions. for instance, more fluent realizations may sometimes be more ambiguous with respect to a formula, compromising faithfulness @xcite . to use a well-worn example, everyone loves someone can be seen as a correct realization of ∀x(p erson(x) → ∃y(p erson(y) ∧ love(x, y))), but the sentence is ambiguous, also allowing for the more specific interpretation that there exists someone who is loved by everyone (i.e., with the scope of the quantifiers reversed). in our work, we use a deterministic procedure for generating text from formulae, allowing us to hold faithfulness constant in order to address issues of comprehensibility and fluency. our work also addresses the question of which human evaluation task is appropriate for a given system @xcite , proposing a novel evaluation using natural language inference (nli; @xcite . in outline, the main contributions of this paper are the following: i) we analyze the quality of the translations of the fol-to-text system presented in @xcite via a human translation quality assessment. ii) we exploit the outcomes of the quality assessment to develop the improved system lola. iii) we present the results of a comprehensive automatic and human evaluation of lola. ## related work although receiving far less attention than other tasks, generating nl text from (logically rich) meaning representation (mr) formalisms has a relatively long tradition in nlg, with approaches ranging from rule-based @xcite @xcite to statistical @xcite and neural models @xcite . several mrs have been the focus of the task: logic-based (e.g., description logic @xcite , fol @xcite , and discourse representation structures @xcite ), graph-based (e.g., abstract meaning representation (amr; @xcite ), and formal languages (e.g., sparql @xcite ). some of these mr formalisms are much simpler than fol. for example, amr has less descriptive power @xcite , whereas datasets such as geo-query @xcite and robocup @xcite , used in, e.g., @xcite , omit logical operators and variable binding. for these reasons, we select fol as our formalism, incorporating different types of formulae and defining a concept of well-behavedness (see §4.2) to characterize those best suited for logicto-text translation. apart from @xcite , closest to our work are the following approaches. @xcite considers the problem of logical form equivalence. @xcite present a rule-based approach to generate text from fol with some syntactic optimizations. @xcite propose an algorithm based on dynamic semantics for a specific class of formulae. @xcite use background axioms to simplify a fol formula. @xcite generates multiple paraphrases from an input formula. @xcite is one of the few logic-to-text approaches using a sequence-to-sequence framework. @xcite generate explanations from a well-defined logical formalism in the context of human-robot dialogue. a common thread in most of this work is the absence of (proper) evaluations. in particular, as in @xcite , the proposals in @xcite , @xcite , @xcite , and @xcite do not include any attempt at systematic evaluations. in @xcite , there is a mention of a very preliminary inspection of the paraphrases generated, with pointers for improving the evaluation left for future work. @xcite make an effort to move beyond standard metrics proposing an automatic evaluation based on recognizing textual entailment and present an informal analysis of some generated sentences. however, a proper human evaluation is missing. @xcite do not evaluate their system using automatic metrics. yet, they perform a human evaluation based on three dimensions and statistically analyze the results. we aim to build a system that, given a logical formula, will produce effective texts (i.e., optimally helpful to the needs of the user) @xcite , thus, carrying out proper evaluations is one of the main focuses of our work. the abstract syntax defines functions for several logical constructs, while concrete syntaxes are for-mulated to generate fol linearizations in six nls. in addition to this, the system performs some coreto-extended ast manipulations (e.g., flattening, aggregation, in-situ quantification, verb negation, and reflexivization) to improve fluency. figure 3 in appendix a shows a graphical visualization of ranta (2011)'s system. the system can parse all wellformed fol formulae without identity @xcite , containing unary and binary predicates and bound variables. grade grinder corpus the grade grinder corpus (ggc; @xcite is a corpus @xmath0.5m fol translations (correct and incorrect) of ca. 300 sentences made by 55k students answering exercises in @xcite . each nl sentence can have multiple (logically equivalent) correct answers. we select just the portion of answers that are marked as correct and filter the formulae that are not parsable by @xcite 's system (i.e., formulae with time stamps, mathematical operators, 3and 4-ary predicates, the identity symbol, and more than 100 characters). this yields around 5, 500 formulae. random generator in ggc, formulae are understandable by humans and have corresponding sentences that are semantically and pragmatically acceptable. however, it might not be representative of the space of all possible formulae. therefore, we additionally create a tool that generates a random fol formula in the space of all possible formulae for a given domain lexicon. ## assessing ranta (2011) to judge the quality of ranta (2011)'s translation system, we set up a translation quality assessment (tqa; @xcite . a group of human evaluators was asked to analyze a list of english translations from fol formulae, generated by ranta (2011)'s system. specifically, we were interested in receiving feedback on three dimensions: (i) faithfulness (i.e., whether the generated text conveys all and only the information of the input formula), (ii) comprehensibility (i.e., whether the generated text is clearly understandable by the evaluator), and (iii) fluency (i.e., whether the generated text is grammatically accurate and natural-sounding). the evaluation dimensions, especially faithfulness and comprehensibility, are not entirely independent of each other. nonetheless, they possess their own traits that we wanted to assess separately. a problematic point is establishing faithfulness to an underlying formula in presence of an ambiguous or incomprehensible sentence. to mitigate this problem, in the nli task introduced for the human evaluation (see §6.2), we gave participants the opportunity to signal text that is ambiguous or incomprehensible. a total of 10 participants (master students, 4 males and 6 females, with a median age of 24.0 years, @xmath1.2) with sufficient knowledge of logic and proficiency in english foot_1 voluntarily participated in the study. setup evaluators were shown batches of 25 formula-translation pairs consisting of (i) a random selection of 10 formulae extracted from the parsable portion of the ggc corpus and their ranta (2011)'s translations, (ii) 10 randomly generated formulae and their @xcite 's translations, and (iii) 5 filler formulae with incorrect translations created manually. all participants saw the same 5 filler items. the purpose of the fillers was to verify the participants' knowledge of fol. were at least 2 out of 5 filler items not identified as such by a participant, their survey response would be ignored entirely in the analysis. none of the participants was omitted by this criterion. see appendix g for details on the construction of the fillers and to ensure coverage, each participant was presented with a different batch of experimental items. 3 they were required to judge formulatranslation pairs under the three dimensions mentioned above. in particular, for each pair, they had to answer a polar question on the translation's faithfulness with the original formula, and rate on a 5-point likert scale the translation's comprehensibility and fluency. moreover, the evaluators were asked to perform full post-editing @xcite ) on the translations. the instructions given, the questions asked, and one example batch of experimental items can be found in appendix g. ## lola based on the post-edits we received in the tqa described in §4, we developed lola (system for translating between logic and language), a new logic-to-text system that keeps ranta (2011)'s original system as its backbone but improves it by extending its algorithm. in particular, lola implements the first two categories of post-edits derived from the results of the tqa, i.e., core-to-extended ast-like manipulations and logic-based simplification, leaving out the third one (stylistic paraphrases) for future work. the first class of improvements extends the list of @xcite 's core-to-extended ast conversions with some additional optimizations. see appendix c for details on their implementation. the second class of improvements manipulates an input fol formula through the application of logical equivalence laws, based on partee et al. @xcite , optimization original formula ranta (2011) translation post-edit moving the negation inward ¬(∃x)f rontof (x, a) it is not the case that there is an element x such that x is in front of a. there is no element in front of a. ## evaluation to assess the quality of fol to nl translations of lola, we set up a thorough comparative evaluation experiment. we compared the translation quality of three different systems: (i) a base-line generating near-literal translations of formulae, which is ranta (2011 lola outperforms the other two systems on all metrics. however, in the context of logic-to-text generation, the results of metrics based on n-gram overlap vs. metrics based on semantic similarity should be interpreted differently. the texts that we are comparing (i.e., ggc ground truth references and texts generated by the three systems) differ considerably in their structural realization, while keeping the same underlying meaning (i.e., they are paraphrases). this is due to the fact that the ggc ground truth references explain the logical formulae, which, in turn, are given as input to the three systems that operate in deterministic ways, ensuring faithfulness of the output texts. therefore, we expect the results of metrics based on semantic similarity to be comparable across the three systems. on the contrary, we should notice more variance with metrics based on n-gram overlap, since they are more reliant on the surface structure of the texts. nevertheless, bertscore is the only semanticsbased metric that is close to following the expected behavior. this might be an additional indication that neural language models are not capable yet to capture deep semantics of nls, but are still biased towards morphosyntactic realizations @xcite . on the other hand, we can observe substantial variance in the results involving n-gram-based metrics. all these metrics favor lola, which apparently creates texts structurally closer to the original ggc ground truth references. ## future work we see two main areas for further investigation. first, we will examine to what extent our approach can be scaled up to include fol with identity by en-hancing the generator, the logical equivalence laws, and crucially, the optimization operations that were applied to the sentences generated. second, we will try to implement the list of linguistic improvements that emerged from the tqa (see ## conclusion we conducted a human tqa on the faithfulness, comprehensibility, and fluency of ranta (2011)'s translations. we implemented part of the results to build lola, an enhanced version of @xcite 's fol-to-text system, which optimizes the input formula when generating text. we evaluated lola against a baseline and ranta (2011)'s original system, performing both automatic and human evaluations. our results suggest that ranta (2011)'s framework, once adequately enhanced with logical equivalence laws, lends itself well to generating nl translations of fol formulae. furthermore, the results indicate the inappropriateness of current standard automatic metrics to evaluate logic-to-text generation, as they focus on assessing different properties than the dimensions relevant for this task. the present work on logic-to-text generation can be potentially beneficial for a variety of applications. paraphrasing systems could profit from the constraints given by logical equivalence laws to generate faithful paraphrases. logic teaching can benefit by incorporating lola in an intelligent tutoring system supporting students and educators. lola could also be the base for a system helping engineers comprehend the convoluted outputs of theorem provers, as literally translating those formulae might result in quite cumbersome text. we hope that this paper will motivate researchers in the broader nlg community to focus more on the issue of generating faithful, comprehensible, and fluent text from logically rich inputs.