Are LLM s Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024)

Abstract: Quantitative reasoning is a critical skill to analyze data, yet the assessment of such ability remains limited. To address this gap, we introduce the Quantitative Reasoning with Data (QRData) benchmark, aiming to evaluate Large Language Models’ capability in statistical and causal reasoning with real-world data. The benchmark comprises a carefully constructed dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models’ quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText. We evaluate natural language reasoning, program-based reasoning, and agent reasoning methods including Chain-of-Thought, Program-of-Thoughts, ReAct, and code interpreter assistants on diverse models. The strongest model GPT-4 achieves an accuracy of 58%, which has much room for improvement. Among open-source models, Deepseek-coder-instruct, a code LLM pretrained on 2T tokens, gets the highest accuracy of 37%. Analysis reveals that models encounter difficulties in data analysis and causal reasoning, and struggle in using causal knowledge and provided data simultaneously. Code and data are in https://github.com/xxxiaol/QRData.

## introduction data is everywhere in our daily life and work, from win-loss charts in sports competitions to insurance databases of national health. we rely on our quantitative reasoning skills to process the data, more specifically, to extract information from the data, draw inferences, and make predictions (bueno de @xcite . what is the average causal effect (ate) from t to y? statistical reasoning t y x1 x2 1 5.60 -0.53 -0.34 0 6.88 -1.74 -1.80 0 3.00 -0.81 -0.20 … … … … … what is a 95% confidence interval for the population mean of y? ihdp.csv causal reasoning reasoning steps 1. understand the data 2. identify the point estimate 3. verify the conditions 4. calculate the margin of error and construct the interval 1. understand the data 2. build the causal model 3. apply propensity score weighting to estimate ate 4. run refutation tests advanced quantitative reasoning with data data analysis like visualization and summarization @xcite . however, little effort has been made to evaluate the more challenging aspects of quantitative reasoning. on the other hand, there are papers evaluating advanced quantitative reasoning abilities of llms @xcite ), but they have not investigated whether llms can apply these skills to real data. in this paper, we focus on two main areas of advanced quantitative reasoning: statistical reasoning and causal reasoning, with examples shown in figure 1 . given a data sheet from sample surveys, statistical reasoning aims to infer the underlying probability distribution, addressing questions like what is a 95% confidence interval for the population mean of y; while causal reasoning aims to understand the causal relationship between variables, addressing questions like what is the average treatment effect foot_0 from t to y. we propose the quantitative reasoning with data (qrdata) benchmark. it requires models to answer a quantitative reasoning question given data sheets as input. our benchmark shares a sim-the csv file ihdp.csv contains data obtained from the infant health and development program (ihdp). the study is designed to evaluate the effect of home visit from specialist doctors on the cognitive test scores of premature infants. the confounders x (x1-x25) correspond to collected measurements of the children and their mothers … ## benchmark construction in this section, we introduce how the qrdata benchmark is constructed. to ensure the quality of our benchmark, we first gather teaching and research resources for quantitative reasoning, and then annotate questions based on the materials. to compare the performance of quantitative reasoning on data and text, we also introduce an auxiliary dataset qrtext for analysis. resource selection. we look for quantitative reasoning resources with the following criteria: 1) the resource is related to quantitative reasoning, especially statistical reasoning and causal reasoning. 2) it contains multiple quantitative reasoning questions with corresponding data and answers. 3) it is open-source and allows adaptation. eight resources are selected based on these criteria, including two books: openintro statistics @xcite and quantitative social science @xcite ; one online learning resource: causal inference for the brave and true 2 ; and five datasets from academic papers: flow cytometry @xcite , ihdp (hill, 2011), jobs @xcite , neuropathic pain diagnosis @xcite , and arctic sea ice @xcite . question collection. we collect multiple-choice questions and numerical questions from these resources, and ensure that the gold answer is unique. for textbooks and online learning resources, we extract the list of data sheets from their websites. if a data sheet is used in exercises, we use the exercise question and its solution as the question and answer; if it is used in the main content, we annotate questions and answers based on the content. for existing causal discovery datasets (flow cytometry, neuropathic pain diagnosis, and arctic sea ice), we collect multiple-choice questions like which cause-and-effect relationship is more likely? a. headache causes groin pain b. groin pain causes headache c. no causal relationship exists following the style of @xcite . for causal estimation datasets (ihdp and jobs), we collect numerical questions of estimating average treatment effect (ate), average treatment effect on the treated (att), and average treatment effect on the control (atc) like the example in figure 2 . more details are in appendix a.1. we also collect data descriptions like the purposes and formats of the data sheets from the resources, and provide them along with questions to models. statistics. we provide the dataset statistics in to check whether our dataset has already been memorized by models in training, we apply the pretraining data detection method min-k% prob @xcite on gpt models and llama-2, and confirm that our dataset is unlikely to appear in their pretraining corpora. qrtext. to separate the challenge of quantitative reasoning from data analysis and analyze whether models master the quantitative reasoning skills, we create an auxiliary benchmark for com-parison called quantitative reasoning with text. questions in qrtext can be answered without data, with an example shown in appendix figure 4 . we collect questions from openintro statistics and two text-based causal reasoning datasets clad-der @xcite and corr2cause @xcite . qrtext contains 290 questions, 100 for statistical reasoning and 190 for causal reasoning. for ease of model comparison, the ratio of statistical/causal questions of qrtext is similar to qrdata. details of qrtext question collection are in appendix a.2. evaluation metric. we report the accuracy by comparing the final answer generated by models with the gold answer. as models may encounter measurement errors in numerical questions (e.g., different causal effect estimation methods will yield minor differences in results), we consider any result @xmath0. ## methods we develop several zero-shot reasoning methods as our baselines. by comparing these methods, we aim to find a good solution to the task of quantitative reasoning with data. table question answering (table qa). the task of table qa asks models to answer a question based on tables. to evaluate if our benchmark can be solved by existing table qa methods, we experiment with a competitive table qa method @xcite . it constructs a table instruction tuning dataset tableinstruct, and finetunes models on this dataset. ## analysis we go deeper into the primary difficulties models face in addressing the task of quantitative reasoning with data. this may provide insights into how to design methods to better solve our task. we also provide an error analysis of gpt-4 in appendix a.6. ## conclusion we propose qrdata, a benchmark of advanced quantitative reasoning with data, to explore llms' statistical and causal reasoning abilities with realworld data. we carefully construct the benchmark with questions from teaching and research materials, and compare natural language reasoning, program-based reasoning, and agent reasoning among a wide range of models. experimental results reveal that models have difficulties in data analysis and causal reasoning. even though powerful llms like gpt-4 have acquired causal knowledge, they can hardly integrate them with the provided data. by proposing this benchmark, we hope to draw attention to the data-based quantitative reasoning ability. we call for more specialized methods to enhance the ability, like training models in the agent reasoning style with data as input, and equipping models with more quantitative reasoning knowledge.