F orest C ast: Open-Ended Event Forecasting with Semantic News Forest (2025)

Abstract: Open-ended event forecasting (OEEF) seeks to predict future events from a given context without being restricted to a predefined scope or format. It plays a crucial role in domains such as risk management and financial decision making. Although large language models show potential for OEEF, existing approaches and datasets often overlook the complex relationships among events, and current research lacks comprehensive evaluation methods. To address these limitations, we propose ForestCast, a prediction pipeline that extracts forecast-relevant events from news data, organizes them into a story tree, and predicts subsequent events along each path. The pipeline comprises four stages: (1) grouping news into event nodes, (2) constructing a news story tree, (3) mining the semantic structure of the tree, and (4) predicting the next event node and evaluating prediction quality. To support this pipeline, we construct NewsForest, a dataset of 12,406 event chains, each representing a chronologically and logically linked sequence of news events. In addition, we introduce a comprehensive evaluation framework that measures both the accuracy and the quality of prediction. Experimental results demonstrate that ForestCast improves the ability of LLMs to forecast events in news data.

## introduction event prediction, the task of forecasting future events based on past and current information, has great potential in applications including policy making, risk management, and financial decisionmaking @xcite . accurate predictions can help decision makers anticipate challenges and seize opportunities @xcite . conventional forecasting approaches include script event prediction @xcite and temporal knowledge graph completion (tkgc) @xcite . these methods are restricted to predicting specific attributes and selecting answers from a finite range @xcite @xcite . however, real-world developments are rarely limited to such specific scopes, and critical information is not always captured by predefined attributes. the text generation capabilities and open prediction space of large language models (llms) make them promising for open-ended event forecasting (oeef). thus, previous work has used llms for such forecasting tasks @xcite . however, oeef presents two challenges. first, existing prediction methods and datasets typically handle massive forecast-related data by simply grouping and summarizing @xcite @xcite , overlooking the complex relationships between events in forecast-related data. second, due to the open-ended characteristic of oeef, existing evaluation methods @xcite cannot reasonably evaluate prediction results. to address these challenges, we introduce forest-cast, a method for the oeef task that leverages llms and captures evolutionary structures within event data to forecast multiple potential evolutionary trajectories. forestcast is built on the newsforest dataset, which highlights key drivers of event progression and uncovers the logical relationships between them. to evaluate the performance of forestcast, we propose an evaluation framework that measures both the accuracy and the quality of the predictions. experimental results demonstrate improvements in both accuracy and quality after fine-tuning with the newsforest dataset, including a maximum pass@1 improvement of 6.00% and a 0.18 increase (on a five-point scale) in average quality metrics. the dataset and the code are available at https://github.com/bitvis2021/forestcast . the contributions of this work are as follows: ## related work this section reviews the literature on news story tree construction and event prediction. ## the forestcast method we propose the forestcast method, which forecasts events based on the organization of evolutionary structures. our method requires users to provide topic-related keywords along with the start and end dates of the news collection period. for example, if a user wants to generate forests for the follow-up events of trump's election campaign in february 2025, the user would specify the keyword (trump, election) and duration (20250201 to 20250228). appendix a.1.4 describes the time complexity of the method and its time cost on a specific hardware setup. we develop an interactive system (see appendix figure 5 ) for the user to provide such specification. the system automatically collects and analyzes relevant news, constructs the story tree, and produces predictions. we detail additional functions of the interactive system in appendix a.1.1. as figure 1 shows, the forestcast method consists of four steps: ## the newsforest dataset the newsforest dataset contains 12,406 news story chains. by masking the event chains, the dataset can be used to fine-tune llms to improve their predictive capabilities. an example of masked event chain is shown in ## evaluation framework evaluating oeef results is tricky because general predictions that do not specify event details are more likely to be true by chance. to address this issue, we evaluate predictions based on both accuracy and quality metrics. accuracy: to measure accuracy, we determine whether an event has occurred with online header content instruction task: please predict the next node in the chain based on the background. input background: in new york, a federal judge kept a lawsuit against trump alive, while trump's legal team pushed for dismissals. judges delayed or recused from trump-related cases. democrats confirmed judges pre-inauguration, and republicans tried to stall trump trials. chain: 1. a federal judge denied throwing out a baseless lawsuit filed against trump, maintaining its validity in court proceedings. 2. in colorado, republican judges dismissed trump's election interference and secret documents cases, potentially setting aileen cannon as a future ag; these rulings could sway the outcome of trump supporter candidate selection in court appointments. 3. judges continue to cancel trump's 2020 election case court deadlines post-presidential win, with a judge shutting down gop challenge months after trump's victory. 4. the judge paused trump-related january 6 cases, citing special counsel's request and concerns about timing after trump's election win and before his inauguration. chosen after joe biden's victory in the us presidential election, a judge paused donald trump's legal challenges to the 2020 election results. this decision came shortly after the formal certification of the election outcome. judge jack smith put the federal election interference case against trump on hold. reject rioters sought charges dismissal unless trump was also convicted, but lost in court, and are sentenced to prison following the capitol riot. the llm decides whether the prediction has actually occurred. if so, the prediction is scored on the following quality metrics. quality metrics: to evaluate the informational value of predictions, we employ llms to assess prediction quality. following @xcite 2024 ing scoring criteria and examples to prompt llms to evaluate the quality of predictions. these five metrics are also used in the evaluation of dataset quality (see section 4.4). note that we use only five of the six metrics. we exclude branch rationality because it is specifically designed for tree structures and reflects the quality of a story tree rather than the quality of predictions. ## experiments we evaluate the effectiveness of the forestcast method through experiments conducted across multiple llms. our goal is to examine whether the forestcast method effectively captures hidden relationships that llms can learn. we created a test dataset for the experiments, which includes both recent hot topics from april 13 to 20, 2025, and long-term popular topics from the past six months, such as the us tariff policy. for these topics, we constructed news story trees with time spans of 20, 30, 40, 60, and 80 days, similar chang'e 6 attracted widespread international attention. after chang'e 6 completed its important mission, it received some international feedback. chang'e 6 successfully collected and returned samples... attracting international attention. chang'e 6 successfully brought back samples... space agencies from many countries expressed congratulations. 5 (very specific) clearly describes main events, identifies key participants, specific actions, results/reactions. provides verifiable details. chang'e 6 successfully brought back samples... nasa administrator bill nelson sent congratulations to china, expressing pleasure at the mission's achievement. to the configuration of the newsforest dataset, all concluding on april 20, 2025. specifically, we reserved a 30-day period to ensure that all true predictions could unfold. to examine whether the newsforest dataset captures implicit real-world associations learnable by llms, we fine-tuned qwen2.5-7b (qwen team, 2024), llama3.1-8b @xcite , and gemma2-9b (gemma team, 2024), using dpofake and dpo-irrelevant datasets. appendix a.4 describes fine-tuning details. we analyze the prediction accuracy and quality of the model at different time lengths before and after fine-tuning. specific results are shown in figure 3 . we observe that llms exhibit a marked upward trend in predictive accuracy as the time span increases. quality metrics show a slight decline with increasing time length. for near-term events, llms tend towards bold yet effective predictions, whereas for long-term events, llms lean towards stable yet unremarkable predictions. ## conclusion this paper proposes forestcast, a pipeline for openended event foresting that organizes large-scale news into story trees, mines semantic structures, and predicts future developments. we also introduce newsforest, a dataset of 12,406 event chains, and proposed an evaluation framework. experiments on multiple llms show that forestcast improves prediction accuracy and quality. this work provides a practical foundation for building more effective event forecasting systems. our results imply that data structures corresponding to visualizations useful to humans are also useful to llms.