Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023)

Abstract: Tabular data is prevalent across various industries, necessitating significant time and effort for users to understand and manipulate for their information-seeking purposes. The advancements in large language models (LLMs) have shown enormous potential to improve user efficiency. However, the adoption of LLMs in real-world applications for table information seeking remains underexplored. In this paper, we investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios. These include the LogicNLG and our newly-constructed LoTNLG datasets for data insight generation, along with the FeTaQA and our newly-constructed F2WTQ datasets for query-based generation. We structure our investigation around three research questions, evaluating the performance of LLMs in table-to-text generation, automated evaluation, and feedback generation, respectively. Experimental results indicate that the current high-performing LLM, specifically GPT-4, can effectively serve as a table-to-text generator, evaluator, and feedback generator, facilitating users’ information seeking purposes in real-world scenarios. However, a significant performance gap still exists between other open-sourced LLMs (e.g., Vicuna and LLaMA-2) and GPT-4 models. Our data and code are publicly available at https://github.com/yale-nlp/LLM-T2T.

## introduction in an era where users interact with vast amounts of structured data every day for decision-making and information-seeking purposes, the need for intuitive, user-friendly interpretations has become paramount @xcite @xcite . given this emerging necessity, table-to-text generation techniques, which transform complex tabular data into comprehensible narratives tailored to users' information needs, have the emergence and remarkable achievements of llms @xcite 2 and 3, respectively. in this paper, we investigate the table-to-text generation capabilities of llms in two real-world table information seeking scenarios: 1) data insight generation @xcite , where users aim to promptly derive significant facts from the table, anticipating the systems to offer several data insights; and 2) query-based generation @xcite , where users consult tables to answer specific questions. to facilitate a rigorous evaluation of llm performance, we also construct two new benchmarks: lotnlg for data insight generation conditioned with specific logical reasoning types; and f2wtq for free-form question answering that requires models to perform human-like reasoning over wikipedia tables. we provide an overview of table information seeking scenarios and our main research questions in figure 1 , and enumerate our findings as follows: rq1: how do llms perform in table-to-text generation tasks? finding: llms exhibit significant potential in generating coherent and faithful natural language statements based on the given table . for example, gpt-4 outperforms state-of-the-art fine-tuned models in terms of faithfulness during both automated and human evaluations. the statements generated by gpt-3.5 and gpt-4 are also preferred by human evaluators. however, a significant performance gap still exists between other open-sourced llms (e.g., vicuna and llama-2) and gpt-* models, especially on our newlyconstructed lotnlg and f2wtq datasets. ## table information seeking scenarios ## experiments in the following subsections, we discuss the three key research questions about adopting llms into real-world table information seeking scenarios. specifically, we explore llms' capabilities for table-to-text generation tasks, their ability to assess factual consistency, and whether they can benefit smaller fine-tuned models. the examined systems for each experiment are discussed in appendix b. ## conclusion this paper investigates the potential of applying llms in real-world table information seeking scenarios. we demonstrate their superiority in faithfulness, and their potential as evaluation systems. further, we provide valuable insights into leveraging llms to generate high-fidelity natural language feedback. we believe that the findings of this study could benefit real-world applications, aimed at improving user efficiency in data analysis.