Leveraging Large Language Models for Building Interpretable Rule-Based Data-to-Text Systems (2024)

Abstract: We introduce a simple approach that uses a large language model (LLM) to automatically implement a fully interpretable rule-based data-to-text system in pure Python. Experimental evaluation on the WebNLG dataset showed that such a constructed system produces text of better quality (according to the BLEU and BLEURT metrics) than the same LLM prompted to directly produce outputs, and produces fewer hallucinations than a BART language model fine-tuned on the same data. Furthermore, at runtime, the approach generates text in a fraction of the processing time required by neural approaches, using only a single CPU.

## introduction data-to-text is a field of natural language generation (nlg) that focuses on converting structured, non-linguistic data into coherent text @xcite . this paper, like many others in the field @xcite @xcite , specifically addresses the challenge of generating text from data expressed as rdf triples that consist of a subject, a predicate, and an object. for instance, one possible textualization of the following rdf triples: (mozart, birthplace, vienna), (mozart, birth year, 1756) is "mozart was born in 1756 in vienna." there are two main approaches to the construction of data-to-text systems: rule-based and neural methods @xcite . rule-based approaches @xcite rely on predefined templates and linguistic rules to transform structured data into text, ensuring high precision and control over the output. on the other hand, neural approaches leverage deep learning models to automatically learn the mapping from data to text @xcite . they offer greater flexibility and produce more natural and varied text, but have limited interpretability, are more computationally intensive and prone to producing hallucinations @xcite . this paper combines these two perspectives on building nlg systems and proposes to use a large neural language model to train (implement) a rulebased system. specifically, we propose a training procedure that processes the training set by asking a large language model to write simple python code that would generate the reference text based on the input data. the generated code is executed to check for syntax errors and whether it produces the correct output. the final result of the training of the system is a single file of python code that is able to generate a textualisation for the input data. although experimental evaluation on the webnlg dataset @xcite showed that our automatically written rule-based system does not achieve the performance of a fully finetuned neural model in terms of bleu or bleurt score, it produces significantly fewer hallucinations and outperforms a non-trivial neural baseline on these measures. moreover, our system is fully interpretable and offers high controllability, as it can be modified by a python programmer if necessary. our approach also does not require a gpu during inference and produces text almost instantaneously on a single cpu. we conceptualize a high-level fixed structure for our proposed system's python code which organises processing according to the set of predicates present in the input triples. it contains two main elements: (1) an (initially empty) list of rules capable of converting a set of triples with particular predicates into text, and (2) a rule selector that processes the input triples and executes the corresponding rules. each rule is a plain python code snippet/subroutine, coupled with with simplistic specifications of the expected input, including the expected number of triples and the list of their expected predicates. the rules are arranged in a simple list. before a rule's code is executed, the input triples are always sorted to match the order of the predicates given in the rule's specification. this allows simpler rules to be written and limits the number of potential errors. the rule selector processes the input triples by extracting their predicates and executing the rule that has the same list of predicates in the specification. if there is no matching rule, the input is split into several parts by a splitting mechanism that aims to minimize the number of splits by applying greedy search. it iteratively searches for a rule capable of processing the largest subset of input triples, executes it, eliminates the already processed triples from the input and repeats the process. if no rule can be found by further splitting, the triples are converted to text by a default rule "{subject} {predicate} {object}". ## training: llm-based rule generation the goal of the training procedure is to populate the list of rules with useful rules capable of producing a fluent and hallucination-free description of the input triples. first, the approach makes a single pass through the training set, writing for each training example a python code capable of producing the reference text (sec. 3.1). the training procedure only analyses instances that are not fully covered by already trained rules (i.e. they cannot be processed without applying the splitting mechanism), which significantly reduces the size of the training set effectively needed to train the system. next, the approach uses a simple mechanism to improve the generalisability of the constructed system (sec. 3.2). the triples from the training set are clustered to discover sets of predicates that are likely to occur together on the input. then, for each likely set of predicates, an artificial training example is constructed by interacting with an llm, and then a standard rule construction procedure is applied. ## summary we presented a new way of training nlg systems for data-to-text problems: we use a large blackbox language model to write fully interpretable python code that is able to generate data textualisation in a fraction of the processing time required by fully neural systems. the experimental evaluation showed that the quality of the generated text is somewhere between that of a few-shot prompted llm and bart finetuned on the same training data, offering an interesting trade-off between computational and training data requirements, interpretability and predictive performance. in future work, we will extend the synthetic data generation to out-of-domain situations. we also plan to include new types of rules, such as rules operating at the sentence level (e.g. adding subordinate clauses).