The Power of Simplicity in LLM -Based Event Forecasting (2025)

Abstract: Event forecasting is a challenging task that requires temporal reasoning over historical data. Although iterative reasoning agents following the ReAct paradigm bring improvements to event forecasting tasks, they also increase the cost of each prediction and bring challenges in tracing the information that contributes to the prediction. In this study, we simplify the ReAct framework into a retrieval-augmented generation (RAG) pipeline. Surprisingly, the RAG outperforms ReAct with only 10% of the token costs. Furthermore, our experiments reveal that structured statistical contexts significantly enhance forecasting accuracy, whereas introducing unstructured semantic information (e.g., news article titles) negatively impacts performance. In-depth analyses further highlight that the iterative reasoning traces impair forecasting accuracy in smaller-scale models but benefit larger models (e.g., 70B) in the event forecasting task. These insights underscore existing limitations in large language models’ temporal and semantic reasoning abilities, providing critical guidance for developing more cost-effective and reliable forecasting systems.

## introduction temporal event forecasting, the capability to anticipate future events based on historical and current data, is crucial across domains such as climate change @xcite , finance @xcite , and policy-making @xcite , where timely and accurate predictions directly influence decision-making and strategic planning @xcite . traditional forecasting approaches predominantly employ statistical techniques such as autoregression @xcite or machine learning-based time-series models @xcite . recent advancements in large language models (llms) have enabled novel approaches in this task, leveraging extensive textual resources to predict international events and their potential outcomes @xcite @xcite . @xcite explored incontext learning for event forecasting, and retrievalaugmented generation (rag) has also been applied and demonstrated the effectiveness of llms in solving this task @xcite . agentic methods, which enable llms to autonomously interact with external knowledge sources and dynamically retrieve information, have become increasingly popular for event forecasting tasks. @xcite proposed a pipeline that incorporates both a reasoning agent and an evaluation agent for time series forecasting. @xcite introduced the mirai benchmark, which includes a contextual information database and a pre-defined api that allows llm agents to interact with and retrieve data from the database. the iterative api-based interaction between the llm agent and the database enables step-by-step planning, allowing the model to rethink its reasoning based on additional information retrieved at each step. however, despite this flexibility, our preliminary experiments show that react incurs substantial token inference costs without yielding more accurate event forecasting predictions compared to rag. as shown in figure 1 , gpt-4o-mini, when paired with rag using the same retrieved data, achieves comparable or superior performance at only about 10% of the inference cost relative to re-act on the mirai benchmark. 1 for visualization, the zero-shot cost is used as the baseline to scale the token costs of rag and react. building on the observation that rag can outperform react with significantly lower token costs, we systematically evaluate its generalizability across a range of llm architectures. additionally, we investigate the factors contributing to the performance differences between rag and react on the mirai benchmark. the contribution of this work is as follows: ## conclusion our investigation revealed that rag can outperform the agentic framework react in the mirai temporal event forecasting task. besides, models achieve higher robust performances when using structured relation distributions or graphs as context, whereas raw news articles, despite their semantic richness, negatively affect forecasting accuracy. in addition, our findings highlight the current limitation of small-scale llms in aligning intricate, semantically rich contexts to specific event predictions, suggesting a need for further research into semantic grounding and structured contextual representation. this research reveals the practical advantages of simplifying event forecasting frameworks, suggesting that strategic, structured data retrieval within rag methods can yield more efficient and accurate predictive systems than the more elaborate react paradigm.