An LLM -Based Approach for Insight Generation in Data Analysis (2025)

Abstract: Generating insightful and actionable information from databases is critical in data analysis. This paper introduces a novel approach using Large Language Models (LLMs) to automatically generate textual insights. Given a multi-table database as input, our method leverages LLMs to produce concise, text-based insights that reflect interesting patterns in the tables. Our framework includes a Hypothesis Generator to formulate domain-relevant questions, a Query Agent to answer such questions by generating SQL queries against a database, and a Summarization module to verbalize the insights. The insights are evaluated for both correctness and subjective insightfulness using a hybrid model of human judgment and automated metrics. Experimental results on public and enterprise databases demonstrate that our approach generates more insightful insights than other approaches while maintaining correctness.

## introduction in an era of data-driven decision-making, the ability to extract insights from complex databases has become increasingly important @xcite . these insights provide meaningful and actionable information derived from raw data. insights are often derived through statistical, analytical, and computational methods. for example, given the database in figure 1 , a valuable insight would be "higher percentages of students eligible for free or reduced-price meals (frpm) correlate with lower average sat scores in reading, math, and writing." insights are essential to make informed decisions. this is crucial in business contexts, healthcare, or scientific research, where better insights lead to more effective outcomes and advancements @xcite @xcite . generating insights traditionally requires significant manual effort. analysts must meticulously preprocess, explore, and refine the data. it also requires specialized knowledge of the domain to analyze and interpret the data, further complicating the task. this makes the process both time-consuming and resource-intensive @xcite . there have been multiple proposals for automating data analysis tasks @xcite @xcite . however, their outputs are not as insightful as humancrafted insights. they often only operate on single tables and require pre-cleaned data or user-defined goals, which limit their applicability to real-world use cases. our approach introduces a novel framework leveraging large language models (llms) to automatically generate insights. however, the process of extracting valuable insights from raw data with llms is non-trivial. first, llms have limited context size and cannot be fed with entire databases @xcite . second, llms still struggle in processing structured data @xcite . because of these issues, we adopt an agentic approach, where external tools (sql scripts, in our case) are used by the llm to handle structured data. while this is a popular solution, existing methods mostly produce "shallow" insights, such as the identification of simple trends or outliers @xcite . as depicted in figure 1 , our idea is to obtain richer insights by first creating high-level questions over the database. those questions are more complex than the insights that can be directly generated by the llm. we exploit the fact that llms are able to split a question down into simpler ones that can this database consists of three main tables: schools, satscores, and frpm. each the generated insights are evaluated using a hybrid approach that combines human judgment with automated metrics, ensuring a rigorous assessment of both correctness and insightfulness. experiments on public and real company databases show that our solution clearly outperforms existing alternatives in terms of insightfulness of the output texts. ## preliminaries we first introduce text-to-sql as it plays a key role in our solution. we then define insights and the criteria for evaluating them. ## problem definition the main objective of this work is the generation of short texts that are both insightful and correct (sections 2.2.2 and 2.2.1). depending on the context, insightfulness and correctness might have different importance. we define the weighting factor α in the resulting function. the objective is to maximize the weighted harmonic mean: by default, we assign insightfulness and correctness the same importance (α = 0.5). ## proposed architecture we generate insights with a 3-step architecture, as in figure 2 , that (1) generates high-level questions and splits each of them into simpler subquestions that are easier to answer, (2) then answers and validates them using sql to (3) finally post-process them into the final insight. ## evaluation our proposed framework was evaluated on both insightfulness (section 2.2.1) and correctness (section 2.2.2) using a combination of human and llm evaluation. ## related work we first position our work w.r.t. the literature on insight generation. we then discuss text-to-sql and how it is relevant to our solution. automatic insights. several models have been proposed for insight generation from tabular data, such as insightpilot @xcite , quickinsights @xcite , openai data analysis (openai, 2024a), or langchain's pandas agent @xcite . these works propose different definitions of insights. some approaches consider insights derived from predefined templates @xcite populated with aggregate measures across subsets of tables. the generation of these insights is done either with a mining framework based on metrics @xcite or using an llm to guide the process @xcite . although this approach offers several strengths, including providing visual data and ensuring factual correctness, it also presents certain limitations. other approaches (openai, 2024a; langchain, 2024b) directly leverage llms to generate code to obtain insights. these methods are often specialized in single-step data analysis tasks (for example, calculating a correlation or doing some regression task) based on very concrete user instructions. all previous approaches are based on one or more of the following assumptions and limitations: text-to-sql. text-to-sql is one of the most popular approaches for enabling natural language interfaces (nl) to databases. it consists of converting nl questions into valid sql queries, where the result of the query is a set of tuples that answer the question. the advent of llms has facilitated the development of complex architectures, such as llm chains and agentic flows, which have significantly enhanced the efficacy of text-to-sql systems @xcite @xcite . our work employs such architectures to facilitate the automatic generation of insights from databases via automatically generated sql queries. while there is evidence that text-to-sql systems struggle with proprietary, large schemas @xcite , there are promising solutions to tackle this problem with prompt compression @xcite . ## conclusions we have introduced a new method to generate insights from multi-table databases. we proposed an architecture that generates interesting questions, breaks them into more easily answerable subquestions, answers them with automatic sql queries, and summarizes all results into a final insight. we evaluated the insightfulness and correctness of our method using human and llm evaluation against three baselines and two variations. we demonstrated that our method achieves better and more consistent results on these two metrics. for future research, the model's evaluation could be extended to encompass a broader range of domains or data sources (e.g., textual documentation using rag, knowledge graphs or internet search). also, there are techniques that might enhance the performance, such as k-shot learning or prompt learning techniques @xcite . finally, an interactive exploration of the insight search space could lead to better outputs with a limited effort, for example by extending the solution to consider user feedback on early results.