Is GPT -4 a Good Data Analyst? (2023)

Abstract: As large language models (LLMs) have demonstrated their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by artificial intelligence (AI). This controversial topic has drawn great attention in public. However, we are still at a stage of divergent opinions without any definitive conclusion. Motivated by this, we raise the research question of “is GPT-4 a good data analyst?” in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data analysts.

## introduction large language models (llms) such as openai's gpt series have shown their strong abilities on various tasks in the natural language processing (nlp) community, including data annotator @xcite , data evaluator @xcite @xcite , etc. beyond nlp tasks, researchers also evaluate the llm abilities in multiple domains, such as finance @xcite , healthcare @xcite , biology @xcite , law @xcite , psychology @xcite , etc. most of these researches demonstrate the effectiveness of llms when applying them to different tasks. however, the strong ability in understanding, reasoning, and creativity causes some potential anxiety among certain groups of people. as llms are introduced and becoming popular not only in the nlp community but also in many other areas, those people in and outside of the nlp community are considering or worrying whether artificial intelligence (ai) can replace certain jobs @xcite . one such job role that could be naturally and controversially "replaced" by ai is data analyst @xcite . the main and typical job scopes for a data analyst include extracting relevant data from several databases based on business partners' requirements, presenting data visualization in an easily understandable way, and also pro-viding data analysis and insights for the audience. this job involves a relatively routine scope, which may become repetitive at times. it also requires several technical skills, including but not limited to sql, python, data visualization, and data analysis, making it relatively expensive. as this job scope may adhere to a relatively fixed pipeline, there is a heated public debate about the possibility of an ai tool replacing a data analyst, which attracts significant attention. in this paper, we aim to answer the following research question: is gpt-4 a good data analyst? to answer this question, we conduct preliminary studies on gpt-4 to demonstrate its potential capabilities as a data analyst. we quantitatively evaluate the pros and cons of llm as a data analyst mainly from the following metrics: performance, time, and cost. specifically, we treat gpt-4 (gpt-4-0314) foot_0 as a data analyst to conduct several end-to-end data analysis problems. the flow of our proposed framework is shown in figure 1 . according to the given question, the model has to identify the relevant tables and schemes in the databases that contain the necessary data, and then extract the data from the databases and organize it in a way that is suitable for figure generation. then, it is required to analyze the data to identify trends, patterns, and insights that can help answer the initial question. since there is no existing dataset for such data analysis problems, we choose one of the most related datasets nvbench @xcite , and add the data analysis part on top. we design several automatic and human evaluation metrics to comprehensively evaluate the quality of the data extracted, charts plotted and data analysis generated. experimental results show that gpt-4 can beat an entry-level data analyst and an intern data analyst in terms of performance and have comparable performance to a senior-level data analyst. in terms of the cost and time of our experiments, gpt-4 is much cheaper and faster than hiring a data analyst. however, since it is a preliminary study on whether gpt-4 is a good data analyst, we conduct some additional experiments and provide fruitful discussions on whether the conclusions from our experiments are reliable in real-life business from several perspectives, such as whether the questions are practical, whether the human data analysts we choose are representative, etc. these results sug-gest further studies are needed before concluding whether gpt-4 is a good data analyst. to summarize, our contributions include: ## conclusions the potential for large language models (llms) like gpt-4 to replace human data analysts has sparked a controversial discussion. however, there is no definitive conclusion on this topic yet. this study aims to answer the research question of whether gpt-4 can perform as a good data analyst by conducting several preliminary experiments. we design a framework to prompt gpt-4 to perform end-to-end data analysis with databases from various domains and compared its performance with several professional human data analysts using carefully-designed task-specific evaluation metrics. our results and analysis show that gpt-4 can outperform an intern data analyst or a junior data analyst, and can achieve comparable performance to a senior data analyst, but further studies are needed before concluding that gpt-4 can replace data analysts. we did not systematically conduct a large number of experiments using online information as well. the reason is similar to the above. the original questions from the nvbench dataset largely depend on the data stored in the database and rarely require additional knowledge. therefore, we leave the design of such open questions to future work.