Large Language Models for Predictive Analysis: How Far Are They? (2025)

Abstract: Predictive analysis is a cornerstone of modern decision-making, with applications in various domains. Large Language Models (LLMs) have emerged as powerful tools in enabling nuanced, knowledge-intensive conversations, thus aiding in complex decision-making tasks. With the burgeoning expectation to harness LLMs for predictive analysis, there is an urgent need to systematically assess their capability in this domain. However, there are no relevant evaluations in existing studies. To bridge this gap, we introduce the PredictiQ benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. We design an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis.

## introduction predictive analysis @xcite involves making predictions about future outcomes based on past data, using statistical modeling, data mining, and machine learning techniques. it is widely used in decision-making across various fields like business @xcite and healthcare @xcite . to facilitate off-the-shelf predictive analysis for users without data analysis expertise, large language models (llms) @xcite @xcite serve as powerful tools, supporting users with coherent and contextually relevant responses. figure 1 shows how users engage with llms by submitting predictive queries based on specific datasets. given the substantial potential of llm-based predictive analysis, a comprehensive evaluation of leading llms is essential. while no studies specifically assess llm-based predictive analysis, related studies @xcite @xcite @xcite @xcite in relevant fields primarily focus on evaluating either the model's outcomes only (e.g., a total sales figure answering a database query) or the generated codes that, when executed, produce these outcomes. however, the former raises scalability issues as llms may struggle to process the entire lengthy dataset due to limited context length. the latter lacks textual explanations, such as justifications for algorithm selection, which constrains its applicability and undermines user trust. this makes existing evaluation efforts insufficient when directly adapted to predictive analysis, where tasks are inherently more complex. llm-based predictive analysis typically requires handling tasks such as data preprocessing, algorithm selection, result interpretation, and so on. these tasks require both textual explanations and code implementations. in practical applications, textual analysis is crucial for explaining algorithm selection and enhancing reliability. moreover, effective alignment between text and code improves user comprehension of llm-generated solutions. thus, textual analysis, code generation, and textcode alignment are integral to the functionality of ai-driven assistants. in this paper, we propose the predictiq (q stands for query) benchmark to evaluate the potential of current llms in predictive analysis. we first collect datasets covering eight common fields, including economics, traffic, and more. diverse datasets form the basis for predictiq to assess llms' performance in predictive analysis. based on the collected datasets, we collaborate with data analysis experts to formulate predictive queries tailored to the datasets. each query is limited to a single, self-contained question that relies solely on the dataset, without requiring external knowledge. we then formulate the data-specific queries and corresponding datasets into prompts, which are provided to llms to generate responses. we propose an evaluation protocol encompassing three domains: textual analysis, code generation, and text-code alignment, focusing on seven key aspects. the responses from the llms are then evaluated by (i) data analysis experts and (ii) llms. the llm whose responses align most closely with those of the human experts is selected as the primary evaluator. in this study, we adopt gpt4turbo, which achieves an average alignment ratio of 90.5% with human experts. predictiq involves 1130 queries from 44 datasets across 8 fields, requiring 300 human hours for query generation and examination. our evaluation on predictiq costs 900 human hours for response evaluation, 72.18 million input tokens, and 20.4 million output tokens. we conduct extensive experiments on predic-tiq with twelve renowned llms. we find that (i) fine-tuning llms on code enhances both text analysis and code generation, occasionally pushing the model beyond its parameter limits. (ii) code generation and text analysis are interconnected processes that shape llms' overall predictive analysis proficiency. (iii) several llms' predictive analysis abilities vary significantly across different fields, with wide margins, and exhibit diversified score distributions. we believe existing llms are far from mastering predictive analysis, both in terms of performance and efficiency. our contributions are summarized as follows. ## preliminary predictive analysis refers to the use of statistical algorithms and machine learning techniques to analyze data patterns, predict future events, trends, or behaviors @xcite . it is widely used in various fields, including finance, marketing, healthcare, and risk management, to anticipate potential risks, identify opportunities, and inform the decision-making process. see figure 1 for an illustration. problem definition. the predictive analysis process with llms is formalized as follows: the input consists of a predictive analysis query associated with corresponding data. a query is a data-specific predictive question that requests future predictions, identifies implicit data patterns, and so on. the data consists of multiple columns and rows. the output includes text, which explains how the prediction is made and justifies the chosen method, and code, which implements the analysis. ideally, the text should provide clear and in-depth analysis based on the query and data. the code should provide related and functionally correct implementation that solves the problem raised by the query. ## the predictiq benchmark this section outlines the details of the predictiq benchmark, covering (i) data collection, (ii) query formulation, and (iii) response evaluation. ## conclusion in this paper, we propose the predictiq benchmark, addressing the gap in evaluating llms' capabilities in predictive analysis. the benchmark provides domain-diversified datasets, data-specific queries, and in-depth evaluation protocols. we evaluate twelve popular llms on the benchmark, highlighting their capabilities and limitations in predictive analysis. from the evaluation, we find the following key insights: (i) fine-tuning llms on code boosts their performance, sometimes exceeding the limits of model parameters. (ii) code generation and text analysis abilities are interconnected, jointly influencing llms' predictive analysis skills. (iii) llms exhibit varying levels of predictive ability across domains. overall, existing llms still have significant room for improvement in achieving both high performance and efficiency in predictive analysis.