R eport GPT : Human-in-the-loop Verifiable Table-to-Text Generation (2024)

Abstract: Recent developments in the quality and accessibility of large language models have precipitated a surge in user-facing tools for content generation. Motivated by a necessity for human quality control of these systems, we introduce ReportGPT: a pipeline framework for verifiable human-in-the-loop table-to-text generation. ReportGPT is based on a domain specific language, which acts as a proof mechanism for generating verifiable commentary. This allows users to quickly check the relevancy and factuality of model outputs. User selections then become few-shot examples for improving the performance of the pipeline. We configure 3 approaches to our pipeline, and find that usage of language models in ReportGPT’s components trade off precision for more insightful downstream commentary. Furthermore, ReportGPT learns from human feedback in real-time, needing only a few samples to improve performance.

## introduction data-to-text generation has been a longstanding problem in natural language processing @xcite . recent advancements in deep learning gave rise to transformer-based models that have achieved state of the art performance @xcite @xcite . within any real-world context, these systems must grapple with hallucinations and omissions caused by the underlying language model. neglecting to address this may lead to the dissemination of misleading or false content. with this motivation we introduce reportgpt, a framework for human-in-the-loop table-to-text generation that consists of a domain specific language and a set of modules that use it as a representation for generating verifiable commentary. human verification of data-to-text generation, while clearly vital, can be difficult and time consuming especially when it involves checking numerical calculations, as the human must perform the calculations in order to verify the output. the reportgpt dsl acts as a proof mechanism, allowing users to effectively verify outputs for relevancy and correctness. our approaches learn efficiently from human feedback using a bayesian updating mechanism and fewshot prompting with language models. our experiments show that usage of language models in our pipeline trade precision for insightfulness. as a result, while there are fewer factual outputs, those that are accurate tend to be more insightful for the end user. ## related works many approaches to data-to-text generation utilize pre-trained, instruction tuned large language models @xcite @xcite . various recent works have proposed improving the factuality and relevancy of these models by grounding the outputs with logical representations @xcite @xcite . @xcite utilize a logical form to represent reasoning paths, which are then ranked and converted to natural language via a surface realization. the ranking uses a bert-base model trained to classify reasoning path and table tuples as correct or incorrect. during generation, a best-first search is conducted using the correct class probability as a saliency score. @xcite pre-train transformers on table to logical form objectives and then fine-tune on table to text objectives. @xcite utilize python programs as intermediate reasoning steps in the chain of thought of language models and demonstrate their effectiveness across 13 mathematical, symbolic and algorithmic reasoning tasks. openai code interpreter @xcite , utilizes python programs in the chain of thought to perform a wide variety of tasks related to data processing and analysis. @xcite propose symbolically grounded generation, where llms are prompted to interleave their output with references to spans of input text. these references are used to reduce the effort of manual verification. reportgpt defines a domain specific language that serves both as an intermediate representation to logically ground downstream text and as a verification mechanism. the dsl contains direct references to the input table, allowing users to quickly verify corresponding textual outputs. going one step further, our pipeline utilizes human feedback as an online learning signal to improve precision. ## reportgpt the reportgpt pipeline consists of a domain specific language for computations over tabular data, as well as a set of modules that interface with it. the modules are: program generation, program execution, surface realization, and human feedback. the pipeline flow is as follows: program generation takes a table as input, outputs a distribution over programs and samples a batch from this distribution. these programs are then executed by the program execution module. the batch of programs and their results are fed into the surface realization module, which outputs a single sentence natural language description corresponding to each program and result tuple. these sentences, as well as their underlying program and result tuple, are shown to the user. the user verifies the correctness and relevancy of each individual sentence by checking its alignment with the underlying program. the human feedback module stores the user selections, both positive and negative, and uses them to update the program generation and surface realization modules. this loop is iterated until the user is satisfied with their set of verified sentences. figure 1 provides an example of the full execution of the pipeline, showing the functionality of each module and intermediate steps. sections 3.1 and 3.2 describe how tables and programs are represented within reportgpt, respectively. sections 3.3-3.6 discuss each of the above-mentioned modules in detail. ## experiments in this section, we describe our experimental design and results. ## results and discussion the results in tables 1 and 2 demonstrate that downstream applications might benefit from different (or possibly hybrid) configurations. a pcfgbased approach increases the acceptance rate but does not necessarily produce commentary that is novel or insightful. in contrast, the chained approach provides higher insightfulness and might be preferred in settings when multi-shot prompting is feasible. lastly, we examine whether the number of operations in the program is associated with the insightfulness of the commentary. this is based on the hypothesis that sophisticated calculations can lead to more novel or non-trivial outputs. ## conclusion motivated by the necessity for human supervision in real world use cases, we introduce reportgpt: a pipeline framework for verifiable human-in-theloop table-to-text generation. reportgpt consists of a domain specific language that enables verifiability, as well as a set of modules that generate and reason about it. we configure 3 approaches to our pipeline, and find a trade-off between precision and insightfulness.