Towards Generating Effective Explanations of Logical Formulas: Challenges and Strategies (2020)

Abstract: While the problem of natural language generation from logical formulas has a long tradition, thus far little attention has been paid to ensuring that the generated explanations are optimally effective for the user. We discuss issues related to deciding what such output should look like and strategies for addressing those issues. We stress the importance of informing generation of NL explanations of logical formulas through reader studies and findings on the comprehension of logic from Pragmatics and Cognitive Science. We then illustrate the discussed issues and potential ways of addressing them using a simple demo system’s output generated from a propositional logic formula.

## introduction the task of generating natural language text from logical form has a long and diverse tradition @xcite , @xcite , @xcite , to name a few early examples). it has been approached from a variety of perspectives targeting different use cases, including providing feedback to students of logic @xcite ), users of logistic software @xcite ), and explaining the output of a reasoning engine @xcite ). however, so far in this domain very little attention has been paid to generating output which is optimally helpful to the user, presumably a nonexpert with little knowledge of formal logic, we aim to build a system which, given a logical formula, will produce an effective natural language explanation. to that end, we discuss challenges which might arise when building and scaling up such a generation system and strategies for addressing these challenges with the user in mind. we aim to conduct studies with potential users (students of logic and/or users of software which operates using logic) to determine what kinds of explanations the system should generate. the rest of this paper is organized as follows: section 2 summarizes related work. section 3 discusses challenges and possibilities related to determining and ensuring effectiveness of the generated output for the users. section 4 illustrates our points by means of a case study on producing explanations of a propositional logic formula. section 5 concludes. ## related work there have been a number of works aimed at generating text from logical form, using either rule-based @xcite ; @xcite ; de roeck and lowden (1986)) or statistical (basile (2015); zettlemoyer and collins (2012); lu and ng (2011)) methods. however, only a few of them explicitly discuss the issues related to the comprehensibility and effectiveness of the generated output. de roeck and lowden (1986) opt for using indentation as opposed to linear text to minimize ambiguity of the generated text, while @xcite 's solution involves bulleted lists. @xcite addresses a related issue, that of generating multiple paraphrases for a logical formula, with a view to subsequently selecting the best one -as many as almost 4500 paraphrases are generated for one formula, but the issue of filtering out ambiguous paraphrases and selecting the best one is left to future work. @xcite apply transformations at the logic level with the aim of making the formula more suitable for generation. studies with human participants to determine what output of nlg systems is preferable have been conducted in other domains. @xcite study the effect of aggregation in the output of an automated tutoring system on the learner and find that what they call functional aggregation, which produces summaries of text relating to the same object or function, benefits the user more than purely syntactic aggregation does. @xcite conduct reading studies to investigate comprehension of referring expressions with a view to improving an nlg system's output. they investigate the interaction between brevity and clarity, and find that brevity is preferred when all alternatives are unambiguous, and otherwise a longer but unambiguous alternative is preferred, which goes to show that there is an advantage to non-brief referring expressions in terms of comprehension. @xcite 's nlg algorithms thus incorporate a trade-off between clarity and brevity. ## user-oriented explanations: challenges and strategies there are a number of questions which need to be answered when developing an explanationproducing system aimed at making the explanations maximally helpful to the user. as @xcite point out, it is not always the case that the inputted logical formula is in a form suitable for generation. some formulas are unnecessarily complex and would therefore tend to produce unnecessarily complex text unless optimised first. to make matters trickier, for expressive logics it is often not decidable whether two logical formulas are equivalent to each other (termed "the problem of logical form equivalence" by @xcite ), so heuristics need to be developed to decide what transformations to apply, and how many, and to determine how suitable a formula is for generation. @xcite assume as a rule of thumb that transformations which will make a formula shorter are likely to also make it easier to comprehend. however, there are some cases where making the formula longer might be warranted, e.g. if that results in a clearer nl explanation. we believe that it would be beneficial to conduct empirical studies on comprehension and preference between text variants generated based on several equivalent formulas in order to develop such heuristics. at the nl generation stage, there are important decisions to be made as well. which phrasings should be used and which ones should be avoided? one of the aspects which can make generation from logic challenging is that the meaning of logical connectives is not always the same as that of their natural language counterparts (grice (1975), moeschler (2017)). for instance, geis and zwicky @xcite argue that an nl conditional is often used as a logical biconditional (for example, if you go to the party, i'll go too is understood to imply that if you do not go, neither will i), while @xcite show that students of logic particularly struggle with the expression of the biconditional as just in case because it has a very different meaning in everyday natural language. in terms of the form of the generated text, there are a number of alternatives which have been used -linear text, bulleted lists and indentation; these presentation decisions will have an effect on the comprehensibility of the generated output. related, what is the optimal amount of aggregation in this context? are there situations where it is preferable not to aggregate? we argue that these questions should be addressed through controlled user experiments where reading comprehension and speed for alternatives is compared along each of these dimensions. we also believe that such resources as the grade grinder corpus @xcite , which contains students' attempts to convert natural language text to fol, can also inform us about which natural language wordings are effective and which ones should be avoided by the generator. both number and nature of incorrect attempts by the students can be used in gaining insights as to what realizations of connectives tend to be misunderstood and what they are misunderstood as. for instance @xcite find that many errors are made when formalizing a sentence in fol requires reordering the antecedent and the consequent. as has been pointed out in related work @xcite ; flickinger (2016)), ambiguity is a challenging aspect of this generation problem: if not controlled for, bracketing or negation scope ambiguity is likely to emerge. @xcite proposes using a parser test to determine whether the generated output can have multiple readings, and select an unambiguous one that way. we believe that that is an effective solution to the ambiguity problem. however, we can imagine a case where, for a sufficiently complex formula, the generator might only produce explanations with multiple readings, or the unambiguous variant is too clunky and difficult to read. in that case, it would be beneficial to know about the respective likelihood of the alternative readings for the user. it could be, for instance, that a reading is identified by the parser which a human is unlikely to consider. with this likelihood information, one could, for instance, select an output which has the fewest possible readings, or only one likely reading. we intend to explore whether such an approximation of likelihood can be obtained using probabilistic parsing @xcite . ## case study: generation from propositional logic we illustrate the above points by means of a concrete example. as a starting point, we built a simple system, which takes a propositional logic formula as input, parses it into a tree structure, optionally applies transformations to the tree, and realizes the output by reading off the tree, left to right. we chose propositional logic as a base case because it is one of the simplest logics in which many of the important discussed in section 3 emerge. consider the following formula, which involves the block language from tarski's world, a software component for teaching logic to students @xcite : at the tree level, we apply a number of meaningpreserving transformations. for any formula, there is an infinite set of formulas equivalent to it, therefore heuristics need to be developed as to what makes a formula a promising candidate for generation. for simplicity, we start with a set of 8 formulas equivalent to (1), obtained by distributing negation, applying de morgan's laws, or reversing the order of the conjuncts and disjuncts. we pass each of these formulas to a simple generator, obtaining 8 wordings. for example, (1) is realized as: which, if any, of the generated versions should be the final output? at this stage, we run a syntactic parser on the generated text to identify how many and what kind of possible readings the generated text may have. we then determine how many distinct parses each of the texts has by computing which of the parses are equivalent to each other. we find that some generated text variants come out as unambiguous, while others have as many as 11 distinct parses. we are not quite done yet. it is worth pointing out that the ambiguities which the parser detects might not perfectly predict what ambiguities might arise for people. for example, (a) can be parsed three ways, with it is not the case having either narrow or wide scope. however, one could argue that the narrow scope reading is a lot less likely. that could be determined using probabilistic parsing. conversely, there could emerge certain mirage ambiguities, where a sentence which is grammatically unambiguous could still be understood multiple ways by the reader, e.g. we can imagine it is not the case that x is smaller than y or the same shape as y being misunderstood as ¬smaller(x, y) ∨ sameshape(x, y) (narrow scope of negation). such cases seem more difficult to foresee. reader studies could be helpful in gaining insight; complexity heuristics could also be introduced with the hope that less complex sentences would be less likely to give rise to such problems. besides ambiguity, there is another dimension along which the effectiveness of the generated text will vary: readability. text length and naturalness both affect readability. interestingly, ambiguity also interacts with readability: there is evidence that ambiguous sentences are processed faster than disambiguated ones, but only when the readers do not anticipate the need to answer in-detail questions about the read text @xcite . methods like aggregation may be employed to improve readability. so, (1) could also be worded as: @xcite x is not a cube, nor is it smaller than or the same shape than y. that phrasing is also no longer ambiguous, which illustrates that aggregation can have an impact on clarity as well as readability. of course, a yet more natural phrasing of ( 1 ) is as follows: it would be challenging to generate (c) automatically given an input formula like (1) since the system would need to have information about what not smaller means. @xcite allow the user to enter background axioms, which partially addresses the problem: the user would need to explicitly indicate the equivalence between not smaller and at least as large, and between not the same and different shape, in order for such output to be generated. in the above, we have experimented with an approach that computes many possible interpretations of each candidate nl text. an interesting avenue for further research is to investigate how such a brute-force approach may be approximated by a set of heuristics, which could then be used in an approach similar to the revision process for nlg @xcite to avoid unnecessary computation: generating output which is estimated to be the best (i.e., most clear and natural), checking these constraints and repeating the process for the next best output if one of the constraints is violated. an open and challenging question is that of generality: if we identify a set of heuristics for a certain class of formulas, how well will they generalize to a different class of formulas or set of predicates? we aim to explore that through controlled experiments. ## conclusion in this paper, we addressed an aspect of the design of an nlg system for explaining the meaning of logical formulas which has often been overlooked: the needs of the user. we discussed questions which we aim to answer when building such a system, such as logical simplifications, paraphrasing and ambiguity, and considered ways in which they can be informed: reading studies with potential users, work with corpora, and insights from cognitive science and pragmatics. we illustrated these questions and potential solutions by means of an example of generating text from a propositional logic formula.