I nsight P ilot: An LLM -Empowered Automated Data Exploration System (2023)

Abstract: Exploring data is crucial in data analysis, as it helps users understand and interpret the data more effectively. However, performing effective data exploration requires in-depth knowledge of the dataset, the user intent and expertise in data analysis techniques. Not being familiar with either can create obstacles that make the process time-consuming and overwhelming. To address this issue, we introduce InsightPilot, an LLM (Large Language Model)-based, automated data exploration system designed to simplify the data exploration process. InsightPilot features a set of carefully designed analysis actions that streamline the data exploration process. Given a natural language question, InsightPilot collaborates with the LLM to issue a sequence of analysis actions, explore the data and generate insights. We demonstrate the effectiveness of InsightPilot in a user study and a case study, showing how it can help users gain valuable insights from their datasets.

## introduction exploratory data analysis (eda) is a demanding task that extracts meaningful insights from data @xcite @xcite . data exploration is a critical step in data analysis. in general, it involves a series of data analysis operations, such as filtering, sorting, and grouping, to discover patterns in data. usually, the process is iterative and interactive, and the user needs to manually explore the data backand-forth to gain insights. this process is often time-consuming and requires considerable domain knowledge and expertise. below, we present an example to illustrate a data exploration process. example. using a student performance dataset from multiple schools (figure 1 ), an education analyst, alice, conducts eda to comprehend trends in math performance. after considerable manual data filtering and sorting, alice captures an upward trend by plotting math scores over time. alice then puts in more effort into manual data filtering for comparing student performance across schools a, b, and c. finally, she observes that both schools a and b illustrate an increasing trend while school c has an outlier in 2020. alice is curious and decides to investigate the outlier. she spends even more time exploring the data back-and-forth, filtering and grouping by various variables until she finally finds that when excluding "take-home" exams, the outlier for school c in 2020 is no longer present. alice notes this finding and concludes that the outlier is caused by a policy change of the exam form in school c in 2020. alice's manual data sifting for insights is effortintensive and time-consuming, highlighting the need for an efficient automated data exploration system to simplify the process. existing solutions. to date, a number of data exploration systems have been proposed in the data management and data mining community @xcite @xcite . in general, these systems leverage a heuristic score function to identify the "best" data exploration sequence (a series of data analysis operations). while these systems show potential, they exhibit key limitations. ➊ user intent ignorance: existing tools are designed for general exploration and often fail to incorporate user intent. for instance, an analyst may be interested in understanding the economics-related factors but receive insights about demographics. ➋ dataset characteristic ignorance: they overlook dataset characteristics, often providing irrelevant insights. for instance, in a flight delay dataset, a correlation between flight delays and weather might yield insights but irrelevant factors like time and weather does not make sense in the context. they fall short in delivering a direct answers to the user question. @xmath0. when @xmath1, most school have an increasing trend while @xmath2. exam @xmath3. intent and generating actions to achieve userspecified goals @xcite . in this regard, we anticipate that llm can be leveraged to drive the data exploration process. however, there are several challenges that impede the adoption. ➌ hallucination: due to the infamous hallucination issue @xcite , llms often generate unreliable contents and are thus not mature for production use. ➍ overwhelming context window: a dataset may contain millions of cells, which is overwhelming for llms to process. ## preliminaries in this section, we introduce the preliminaries of exploratory data analysis. ## problem definition in this section, we define the problem of generating a sequence of data insights to address users' analysis intents. ## evaluation implementation. we implement our tool based on the codebases of quickinsight, metainsight, and xinsight, adding an additional 1.8k lines of c# code and 1.5k lines of javascript code. we use "gpt-3.5-turbo" as our language model and "textembedding-ada-002" is used to generate embeddings. both models are provided by openai. ## discussion action-wise performance. the efficacy of insight-pilot, an automated data analytics tool, hinges on the comprehensive design of each action and its accurate execution. while we introduce four actions rooted in common data analysis techniques, it is vital to note that insightpilot's innovation is not tied to specific action designs. instead, its uniqueness lies in leveraging llm for data exploration. comprehensive assessment. to validate insight-pilot's effectiveness, it is essential to evaluate it across diverse real-life datasets and dimensions, such as keyword preservation. nonetheless, in-sightpilot often produces open-ended responses, making manual evaluation crucial for assessing answer quality. these hurdles make it challenging to efficiently evaluate insightpilot's performance in a comprehensive manner. an llm-based evaluation framework could potentially streamline this process @xcite . ## conclusion in this paper, we introduce insightpilot, an llmempowered automated data exploration system. by seamlessly integrating llm with state-of-the-art insight engines, insightpilot streamlines data analysis into a coherent exploration sequence. effective for real-world datasets, it allows users to derive insights via natural language inquiries. insightpilot equips even non-technical individuals to benefit from data analysis, bolstering efficiency and datadriven decision-making.