Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling (2024)

Abstract: Predicting future international events from textual information, such as news articles, has tremendous potential for applications in global policy, strategic decision-making, and geopolitics. However, existing datasets available for this task are often limited in quality, hindering the progress of related research. In this paper, we introduce a novel dataset designed to address these limitations by leveraging the advanced reasoning capabilities of large-language models (LLMs). Our dataset features high-quality scoring labels generated through advanced prompt modeling and rigorously validated by domain experts in political science. We showcase the quality and utility of our dataset for real-world event prediction tasks, demonstrating its effectiveness through extensive experiments and analysis. Furthermore, we publicly release our dataset along with the full automation source code for data collection, labeling, and benchmarking, aiming to support and advance research in text-based event prediction.

## introduction accurate prediction of future international events is essential for effective decision-making in international relations, global strategy, and security policy @xcite . however, the dynamic nature of international relations, with its evolving conflicts and cooperation between multiple countries, presents a significant challenge for accurate prediction @xcite . recent global events, such as ongoing regional tensions and shifting economic dynamics among various countries, further complicate these challenges for policymakers and analysts. these complexities highlight the importance of developing reliable machine learning models capable of predicting changes in international relationships and the subsequent events they could trigger. existing approaches for text-based international events prediction like @xcite have heavily relied on datasets like the global database of events, language, and tone (gdelt) @xcite due to their comprehensive coverage and ease of use. by providing a comprehensive archive of global news and extracting relationships between subjects like countries, these datasets can be leveraged by machine learning models to tackle predictive tasks @xcite . however, they have notable limitations: gdelt frequently fails to capture complex interactions involving multiple countries, typically focusing on bilateral relationships. this simplification overlooks the intricate multilateral relations that are critical for understanding international dynamics. as analyzed in our study (detailed in @xcite in section 3.1), this limitation becomes evident, highlighting the need for more comprehensive datasets. despite its significant contributions as a seminal work in the field, these limitations highlight the need for a more reliable and extensive dataset. in this paper, we introduce worldrep (world relationship and event prediction), a new dataset designed to overcome existing limitations by leveraging the advanced reasoning capabilities of large-language models (llms) @xcite . to ensure reliable and efficient results, we developed a structured scratchpad @xcite with a self-correcting mechanism, incorporating a verification-correction pattern for annotations. this designed scratchpad allows for step-by-step reasoning, which helps in accurately identifying countries involved in multilateral events. the advanced contextual understanding of these models also ensures precise labeling of international relationships. to validate the quality of worldrep, we conducted extensive experiments involving multiple steps. first, we obtained annotations from domain experts in international relations and political diplomacy. we then compared the number of extracted countries in our dataset to those identified by the experts, finding a high level of agreement. we also measured the alignment of our labels with the experts' labels, ensuring the consistency and reliability of our annotations. next, we trained predictive models on worldrep and evaluated their performance using the expert-provided labels as the test set, achieving strong accuracy and f1 score. leveraging these results confirming the reliability of our annotation process, we automated the labeling process and created a benchmark for predicting future international relations. in summary, we present a comprehensive and reliable dataset that captures complex multilateral interactions. using advanced llms, we accurately annotate relationship labels with a structured prompt design. we make both our dataset and the full automation source code publicly available to sustainably support further research and development in text-based event prediction and international relations. we believe this contribution will significantly advance the field by providing a solid foundation for future studies and applications. ## dataset construction constructing a reliable dataset for research to predict future geopolitical events involves addressing several key challenges due to its complexity and ambiguity. we focus on two main enhancements compared to existing datasets such as gdelt: 1) capturing multilateral relations and 2) improving the accuracy of relationship labeling. our dataset construction process is designed to reflect the complexity of multilateral international relations more accurately. after collecting data, the process involves two main stages, multi-subject extraction and relationship score labeling, as illustrated in figure 1 . by following these steps, worldrep captures the intricate nature of global events and provides high-quality labels for research in predicting text-based international relations. the details of each step are explained below. ## dataset quality evaluation to demonstrate the quality of worldrep, we conducted several experiments, including an evaluation against domain expert labels. our evaluations in this section focused on the reliability of the labels and the validity of the designed prompting system. ## future event prediction in this section, we provide a benchmark for text-based prediction of future relationships between countries. predicting future international relations based on text data is a highly challenging task due to the complexities involved in modeling such events and the high cost of gathering quality-labeled datasets. in the previous section, we demonstrated the superiority of our labeling approach through comparisons with domain expert labels. building on this foundation, we now leverage our automated labeling system to efficiently label new data, replacing the prohibitively expensive expert labeling process. this approach allows for flexible and timely updates of worldrep while maintaining high data quality. based on the results of this labeling process, we provide a benchmark using the latest large models capable of performing this task. we hope that this benchmark will serve as a starting point for further modeling advancements in predicting future geopolitical events. ## conclusion in this paper, we introduced worldrep, a novel dataset designed to predict future international events from textual information, leveraging the advanced reasoning capabilities of large-language models (llms). worl-drep addresses the limitations of existing datasets by capturing complex multilateral relations and providing high-quality labels validated by domain experts. extensive experiments demonstrate the reliability and effectiveness of worldrep in real-world event prediction tasks. furthermore, we established a benchmark for future event prediction, highlighting the potential and challenges of using llms for this purpose. we believe that worl-drep and the automated labeling system will advance research in text-based event prediction and international relations, providing a solid foundation for future studies and applications.