C a LQ uest. PT : Towards the Collection and Evaluation of Natural Causal Ladder Questions in P ortuguese for AI Agents (2025)

Abstract: Large Language Models (LLMs) are increasingly central to the development of generative AI across diverse fields. While some anticipate these models may mark a step toward artificial general intelligence, their ability to handle complex causal reasoning remains unproven. Causal reasoning, particularly at Pearl’s interventional and counterfactual levels, is essential for true general intelligence. In this work, we introduce CaLQuest.PT, a dataset of over 8,000 natural causal questions in Portuguese, collected from real human interactions. Built upon a novel three-axis taxonomy, CaLQuest.PT categorizes questions by causal intent, action requirements, and the level of causal reasoning needed (associational, interventional, or counterfactual). Our findings from evaluating CaLQuest.PT’s seed questions with GPT-4o reveal that this LLM face challenges in handling interventional and relation-seeking causal queries. These results suggest limitations in using GPT-4o for extending causal question annotations and highlight the need for improved LLM strategies in causal reasoning. CaLQuest.PT provides a foundation for advancing LLM capabilities in causal understanding, particularly for the Portuguese-speaking world.

## introduction we are witnessing the massive use of large language models (llms) in the development of generative ais across a wide range of domains, including healthcare, legal decision-making, and customer service. some researchers and commentators have speculated that these tools could represent a decisive step towards machines that demonstrate 'artificial general intelligence' @xcite . however, on the path toward artificial general intelligence-which is purportedly being approached by modern llms like @xcite , gemini @xcite , and claude (anthropic, 2023)-the ability to understand cause-and-effect relationships and engage in causal reasoning is essential @xcite ). in @xcite , pearl proposed the "ladder of causality" to categorize different levels of causal thinking. the first rung, associational, consists of detecting correlations and patterns in observed data. llms already excel at this from their pre-training data. but in the higher pearl's hung -in which it is required to understand the effects of actions and interventions on a system (interventional rung), and imagining and reasoning about hypotheticals and alternate realities (counterfactual rung), in the best case, we need to evaluate how and whether llms have abilities to reason about these situations. @xcite affirms that "these transformative developments raise the question of whether these machines are already capable of causal reasoning: do llms understand causality?". in this regard, we need to provide a set of natural causal questions to increase the capabilities of llms in interventional and counterfactual situations. however, there is a lack of a comprehensive collection of causal questions of this kind in previous works, even for high-resource languages, such as english language. existing causal datasets mainly focus on artificially crafted questions and have zero or limited coverage of natural human questions, not capturing pragmatic nuances and linguistic diversities @xcite ). the portuguese language, despite being the 6th most spoken language in the world with around 270 million speakers, is considered a low-resource language @xcite ) and this lack of datasets and golden standard collection for causal reasoning is even more critical. to date, there is no known benchmarking dataset that includes natural causal questions in portuguese. in this work, we propose the development of calquest.pt 1 , a dataset comprising more than 8,000 natural causal questions in portuguese, collected from public sources and produced by humans in interactions with other humans and software systems. calquest.pt is constructed based on a three-axis taxonomy, also proposed in this work, designed to capture the intent and action requirements in causal reasoning chains and the three rungs of causality defined by pearl @xcite ). we argue that the proposal of calquest.pt, addressed here, is promising, as it will allow the evaluation and training of ai agents to identify when to apply cause-and-effect knowledge or reasoning (axis 1: "causal/non-causal"); to identify the requested action according to the interlocutor's intent (axis 2: "action class"), and finally, to identify the level of reasoning needed by an ai causal solver (axis 3: "causal reasoning" -associational, interventional and counterfactual). an additional contribution of this work is the annotation methodology, which follows a human-in-theloop approach. we evaluating the seed questions of the calquest.pt using the llm gpt4o with two prompt strategies and the findings indicated that gpt-4o struggles to assess the type of reasoning required for causal questions (particularly interventional questions) and to recognize the need to identify cause-and-effect relationships between two variables or events (relation-seeking questions) and the effect of a cause (effect-seeking questions). these results did not support the indiscriminate use of gpt-4o to extend annotation to additional natural questions of calquest.pt. ## related works for the english language, we have datasets with completely artificially generated causal questions, such as wiqa @xcite , head-line cause @xcite , glu-cose ( mostafazadeh et al. (2020) ), cladder @xcite ) and corr2cause @xcite . the datasets e-care @xcite ) contain some natural questions human-to-human and human-to-searchengine, however, these bases do not contain questions between humans and llms, due to having been proposed before the explosion in popularity of llms. especially, @xcite proposes the cladder, a database developed artificially through a causal inference engine, which calquest.pt source code and datasets processes queries, graphs, and other information available in questions classified in the ladder of causality of pearl. recently, @xcite propose the causalquest database containing natural causal questions in their entirety, collected from interactions between humans (humanto-human), between humans and search engines @xcite and between humans and lange language models (human-to-llms). this dataset seeks to meet the need for natural question bases of a causal nature and the need for question bases aimed at llms, which have very particular characteristics, such as the length of each question, which can exceed 100 words per question. the authors argue that the structure of the questions formulated, scenarios, conditions, and examples may be used to improve understanding of llm and optimize its results in causal reasoning for the portuguese language, no studies are addressing the construction of a dataset containing natural causal questions, as well as the various taxonomies for causality, at least to the best of our knowledge to date. this fact already corroborates the importance of this work, as it provides the portuguese language computational processing community with a basis for evaluating llms in causal reasoning. ## calquest.pt data collection and annotation to guide the development of a causal question dataset in portuguese, we defined a threeaxis taxonomy for causality inspired in @xcite and @xcite . we then gathered a total of 8,041 natural questions from databases and repositories containing humangenerated queries, which we used to create our gold standard collection through a human-in-the-loop approach. ## conclusion this work presents an unprecedented proposal for a collection of causal questions, produced by humans in portuguese -calquest.pt, which aims to serve as a basis for evaluating and training ai agents to identify when to apply cause-and-effect knowledge or reasoning, to identify the requested action according to the interlocutor's intention, and finally, to identify the level of reasoning needed by an ai causal solver (rungs associational, interventional and counterfactual). we then proposed a three-axis taxonomy and an annotation methodology, which follows a human-in-the-loop approach. calquest.pt will, therefore, serve to promote studies of ai agents with the capacity for causal commonsense reasoning in portuguese, considered a low-resource language. we evaluated the llm gpt4o in the classification of seed questions from calquest.pt, according to our three-axis taxonomy, and the findings indicated that gpt-4o struggles to assess the type of reasoning interventional and cause-and-effect relationships. these results did not support the indiscriminate use of gpt-4o to extend annotation to additional natural questions of calquest.pt. in future works, we plan to ex-plore other llms, like open source llms -llama, gemma e phi, and fine-tuning processes to enhance results. the variation of examples in the few-shot learning prompt strategy will also be a focus of future investigations, alongside efforts to measure the consistency, repeatability, and reproducibility of llm responses.