Reasoning and Tools for Human-Level Forecasting (2024)

Abstract: enter abstract here

## introduction forecasting is an essential tool today, playing a critical role in government, corporate, and personal decision-making. weather forecasting provides essential information for agriculture, natural disaster preparedness for governments, and travel plans for individuals. during the covid-19 pandemic, lockdown policies were largely determined by forecasts, which were required to be sufficiently accurate due to their global impact @xcite . forecasting methodologies fall into two main categories @xcite : statistical and judgmental. statistical forecasting leverages time-series modeling and excels with abundant data under stable conditions. conversely, judgmental * authors contributed equally to this work. forecasting, which we refer to simply as "forecasting," typically relies on human expertise, integrating historical data, domain knowledge, and intuition to make predictions, and is particularly useful when data are sparse or conditions are volatile. by nature, forecasting requires not only accuracy but also the ability to continuously adapt to dynamic data streams. this is where traditional lms often struggle: timely data updates may cause predictions to change considerably and past data to be irrelevant. ## related work information retrieval reliable and accurate predictions are largely dependent on the information available to the predictor. this is especially the case of lms, which are trained on data preceding a knowledge cutoff and have been shown to perform better with information retrieval @xcite . language models model the likelihood p θ (y i |x, @xmath0. retrieval-augmented generation (rag) @xcite proposes augmenting this approach with non-parametric memory, i.e. retrieving the top-k text documents z via p η (z|x) and conditioning the generator on the retrieved passages, p θ (y i |x, z, @xmath1. in a forecasting context, rag enables us to search for relevant documents z that may contain timely information about the forecasting task x not present in the training data. prior approaches to llm forecasting @xcite propose using neural networks to automate prediction in prediction markets. while language models can be trained to improve their performance on forecasting tasks, their accuracy remains significantly below those of human experts. current methods aim to improve the accuracy of llm forecasting by fine-tuning and scratchpad prompting @xcite @xcite or ensembling @xcite to first approach human-level forecasting. concurrent work @xcite benchmarks llms' forecasting capabilities using the gleangen prediction market, an internal tool at google. however, this approach did not accurately reflect real human crowd prediction distributions, and it relied on palm2 @xcite , which was suboptimal than gpt models. we propose a zero-shot tool-usage llm framework without costly fine-tuning and laborious tedious scratchpad format prompting. ensembles leveraging multiple llm agents has demonstrated strong performance on a variety of tasks, and improve performance beyond that of a single agent @xcite . recent work in tool learning has implemented task planning and execution with separate agents @xcite . llm forecasting in particular has relied on the wisdom of crowds effect, and has shown that taking ensemble sizes up to 36 outperforms any individual forecasting agent @xcite . we propose bridging this gap with a hierarchical structure to facilitate cooperation between highlevel reasoning and low-level execution agents, and demonstrate that a small ensemble suffices for human-level performance. ## reasoning and tools for forecasting forecasting is a complex task solving environment, for which we would like to leverage a frozen lm p θ as reasoning. successful forecasting agents rely on the most up-to-date information, and accordingly operate as agents that collect observations o @xmath2. the observation space o is natural language, as collected from the prompt itself or information on the internet. the agent's actions are distributed according to a t ∼ π(a t |c t ), where c @xmath3, a 1 , . . . , o t-1 , a t-1 ) is the context to the agent. our proposed approach π satisfies the following criteria: