MURMUR : Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation (2023)

Abstract: Prompting large language models has enabled significant recent progress in multi-step reasoning over text. However, when applied to text generation from semi-structured data (e.g., graphs or tables), these methods typically suffer from low semantic coverage, hallucination, and logical inconsistency. We propose MURMUR a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. MURMUR is a best-first search method that generates reasoning paths using: (1) neural and symbolic modules with specific linguistic and logical skills, (2) a grammar whose production rules define valid compositions of modules, and (3) value functions that assess the quality of each reasoning step. We conduct experiments on two diverse data-to-text generation tasks like WebNLG and LogicNLG. The tasks differ in their data representations (graphs and tables) and span multiple linguistic and logical skills. MURMUR obtains significant improvements over recent few-shot baselines like direct prompting and chain-of-thought prompting, while also achieving comparable performance to fine-tuned GPT-2 on out-of-domain data. Moreover, human evaluation shows that MURMUR generates highly faithful and correct reasoning paths that lead to 26% more logically consistent summaries on LogicNLG, compared to direct prompting.

## introduction data-to-text generation @xcite @xcite @xcite @xcite is the task of generating fluent, faithful, and consistent summaries of semi-structured data. recent works have introduced different data-to-text generation tasks 1 supporting code available at https://github.com/ swarnahub/murmur summary-1: reinhold roth had one win in his 250cc career. summary-2: reinhold roth's best finish in the 250cc class was a second-place finish in 1989. summary-1: reinhold roth scored the least number of points in the year he was on the suzuki team. ## definitions: reasoning step and path a reasoning step is a triple (m, x , y) where a module m performs a certain skill by conditioning on an input x to generate an output y. for example, in fig. 2 , the module argmin takes a table and a column (points) as input and outputs the row with the minimum points. a reasoning path is defined as a sequence of such reasoning steps {(m i , x i , y i )} r @xmath0. fig. 2 shows an example of a reasoning path, represented as a nested structure. it consists of three reasoning steps for three modules (argmin, hop, and eq). the argmin module outputs table -> row value function reasoning step 1 reasoning step 2 grammar production rule: row -> string output data type: bool true reasoning step 3 grammar production rule: string -> bool eq{hop{argmin{all_rows; points}; team}; suzuki} reinhold roth scored the least amount of points in the year he was on the suzuki team. final summary surface realization eq suzuki reasoning path table topic: reinhold roth figure 2 : illustration of murmur generating a reasoning path and then converting it into a logically consistent summary, supported by the input table. the reasoning path consists of three reasoning steps. at each step, murmur chooses a set of plausible modules (according to a grammar) and then selects the best module, with the best input according to a value function. the output generated at each step serves as the input to the next step. the row in the table with minimum points, which is the input to the next module hop that selects a column from that row. murmur generates textual summaries by constructing such reasoning paths that are then converted to the final outputs through a surface realization module, as shown in fig. 2 . 4 ## murmur approach murmur consists of four components: (1) a set of modules, (2) a grammar, (3) value function(s), and (4) a search algorithm that brings all the previous three components together. the search algorithm constructs reasoning paths by first identifying plausible modules at each reasoning step according to the grammar and then determining the best modules (and their corresponding inputs) with the help of value functions. fig. 2 shows a working example of murmur, in which given an input table, it searches for a reasoning path (of three steps), and finally converts it into a summary. the specifics of murmur's components vary based on the task at hand. as case studies, we consider two datato-text generation tasks: webnlg @xcite , a graph-to-text generation task and logic-nlg @xcite , a complex table-to-text generation task where the goal is to generate logical summaries from salient parts of the table. ## conclusion we presented murmur, a neuro-symbolic modular reasoning approach for data-to-text generation. murmur shows the benefits of building interpretable modular text generation systems by breaking a task down into sub-problems and then solving them through separate modules, without requiring module-specific supervision. it utilizes the power of llms in solving linguistic sub-tasks through incontext learning while delegating the logical subtasks to symbolic modules. murmur generalizes the concept of modules by treating them as functions and defining their behaviors through expected input/output data types and compositions with a grammar (analogous to function compositions).