Beyond Inherent Cognition Biases in LLM -Based Event Forecasting: A Multi-Cognition Agentic Framework (2025)

Abstract: Large Language Models (LLMs) exhibit strong reasoning capabilities and are widely applied in event forecasting. However, studies have demonstrated that LLMs exhibit human-like cognitive biases, systematic patterns of deviation from rationality in decision-making. To explore the cognitive biases in event forecasting, we introduce CogForecast, a human-curated dataset comprising six topics. Experimental results on three LLMs reveal significant cognitive biases in LLM-based event forecasting methods. To address this issue, we propose MCA, a Multi-Cognition Agentic framework. Specifically, MCA leverages LLMs to act as multi-cognition event participants, performing perspective-taking based on the cognitive patterns of event participants to alleviate the inherent cognitive biases in LLMs and offer diverse analytical perspectives. Then, MCA clusters agents according to their predictions and derives a final answer through a group-level reliability scoring method. Experimental results on a dataset including eight event categories demonstrate the effectiveness of MCA. Using Llama-3.1-70B, MCA achieves an accuracy of 82.3% (79.5% for the human crowd). Additionally, we demonstrate that MCA can alleviate the cognitive biases in LLMs and investigate three influencing factors.

## introduction recently, large language models (llms, @xcite such as chatgpt have shown remarkable reasoning capabilities across various applications, including event forecasting. event forecasting @xcite @xcite is a challenging task that aims to predict future developments based on the analysis of background information. basically, llm-based forecasting methods can be categorized into prompt engineering @xcite , retrieval-augmented generation @xcite , instruction tuning @xcite , and llm-based agent methods @xcite . these studies treat llms as objective analysts and contribute significantly to the progression of event forecasting. however, as demonstrated in @xcite and @xcite , llms inherit human-like cognitive biases from human-created data. the cognitive biases are systematic patterns of deviation from norm or rationality in decisionmaking, thus rendering llm-based methods insufficient for objective decision-making. to investigate the cognitive biases in event forecasting, we introduce cogforecast, a human-curated dataset comprising six topics (each with a pair of entities). using a cognitive preference score as the metric, three llms show significant cognitive biases. furthermore, cognitive biases are also observed in agents using domain experts, such as political scholars and analysts, and the final judge in multiagent debate systems (mad, @xcite , as depicted in figure 1 . to mitigate the cognitive biases of llms, we propose mca, a multi-cognition agentic framework for complex event forecasting. the method is motivated by perspective-taking in cognitive theory, which is widely applied in international relations analysis. as illustrated in figure 1 , mca profiles agents as multi-cognition event participants for perspective-taking, facilitating llms in shedding inherent cognitive biases and offering a comprehensive perspective. specifically, mca includes two stages: agent construction and forecasting. in the agent construction stage, mca proposes an automatic agent construction method that clusters historical events and extracts multi-cognition participants, resulting in a large-scale agent collection. in the forecasting stage, given a question, mca dynamically retrieves relevant multi-cognition agents from the agent collection. subsequently, a retrieval assistant collects multilingual, multi-cognition information from news websites and youtube to alleviate information cocoons. based on retrieved information, agents engage in perspective-taking from the viewpoint of event participants, facilitating comprehensive analysis from diverse perspectives. finally, to support objective collective decision-making (cdm), mca clusters agents according to their predictions and derives a final answer using a group-level reliability scoring method. in experiments, we evaluate mca on a challenging forecasting benchmark, including eight categories. mca demonstrates its superiority across four llms, yielding an average accuracy improvement of 4.7% (especially in the "security" category). notably, using llama-3.1-70b as the llm, mca yields an accuracy of 82.3%, surpassing the human crowd's 79.5%. additionally, we demonstrate that mca can alleviate cognitive biases and explore three factors influencing the cognitive biases and prediction performance of llms, including agent profiling, information source, and cognitive certainty. regarding the cdm, we compare various cdm mechanisms, highlighting the sensitivities of dictatorial and debating methods, and demonstrating the effectiveness of our method. our contributions are as follows: ## the cognitive biases of llms in event forecasting dataset to address the dataset gap in assessing cognitive biases of event forecasting, we introduce cogforecast, a human-curated dataset comprising 6 topics (6 pairs of entities t @xmath0, e 2 i ] 6 @xmath1, 218 samples). these entity pairs exhibit significant cognitive discrepancies, including "us-china", "us-iran", "ukraine-russia", "palestine-israel", "south korea-north korea", and "syrian-hts". each sample contains a question and three options, such as: "question: in 2024, the syrian opposition hts succeeded in overthrowing the assad government. will syria gain more freedom and democracy? options: (a) cannot answer; (b) yes; (c) no". given the significant cognitive divergence between e 1 metrics for topic t i , prediction on question q j i is mapped to preference p j i . then, for e 1 i , e 2 i , and neutral option "cannot answer", we calculate their cognitive preference scores as: results as depicted in figure 2 , when employing cot (first row), three llms consistently exhibit a pronounced cognitive preference for e 1 i (blue bar) over e 2 i (gray bar) across all topics. additionally, different llms show varying degrees of cognitive biases, with llama-3-8b exhibiting the most significant biases. using the similar prompt template as cot, we evaluate three kinds of agents: expertprompting (you are an international relations analyst specializing in the analysis of e 1 i -e 2 i relations), spp (a multi-agent system that simulates collaboration among domain experts), and mad (a two-round multi-agent debate system that simulates debates between affirmative and negative sides). however, compared to cot, expertprompting exacerbates the cognitive biases of llms. after incorporating additional experts, spp exhibits minor fluctuations in cognitive bias across dialog rounds (average shift in preference e 1 i is 4.3% across three llms). for mad, while the affirmative and negative sides display different cognitive preferences (averaged preference difference is 11.3% across three llms), the final judge remains the cognitive preferences of llms. to further explore the cognitive biases, we examine three specific types of cognitive bias: in-group bias, premise-induced bias, and confirmation bias. in-group bias refers to the tendency to favor members of specific group over out-group members. experimental results reveal that all llms exhibit a marked preference for western-aligned entities, indicating a pronounced in-group bias. premise-induced bias refers to the influence of contextual premises on model outputs, often exacerbating or reversing cognitive bias. we observe that the inclusion of premises aligned with entity e 1 i amplifies the cognitive bias, while opposing premises may mitigate or even reverse it. confirmation bias refers to the model's tendency to favor information that aligns with its pre-existing cognitive framework. despite some reduction in bias when contradictory premises are introduced, most llms still exhibit a strong preference for the entity that aligns with their inherent stance. detailed analysis is provided in appendix a.4. experimental results on additional llms and opensource twinviews dataset @xcite are provided in appendix a.3 and a.5, respectively.