Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models (2025)

Abstract: The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including ~5K manually annotated questions. We also investigate automatic evaluation methods and propose reference-based techniques as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code plus a public leaderboard are provided to encourage further research, not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking.

## introduction recent research has documented significant pedagogical concerns regarding the proliferation of artificial intelligence conversational interfaces. specifically, it suggests that these systems might be reinforcing superficial learning processes while simultaneously diminishing users' capacities for critical thinking @xcite . in this context, researchers have proposed critical questions generation (cqs-gen) as a method to leverage llms to automatically generate critical questions that systematically expose evidential weaknesses or structural flaws (such as fallacies) embedded within argumentative discourse. the 1 https://github.com/hitz-zentroa/ benchmarking_cqs-gen argument: "i did not use sunscreen yesterday and i was fine, so i don't think you need it today." critical questions: ## previous work the automatic generation of critical questions is a relatively new task, introduced and motivated by @xcite . their work demonstrated that theoretical critical questions, such as those proposed by @xcite , and questions generated by llms are complementary, as they tend to target different aspects of argumentation. this is in line with the observation that cqs can not be restricted to a prefix set @xcite . calvo figueras and agerri (2024) also identified common failure modes in llm-generated critical questions, including: (a) producing overly generic or unrelated questions, (b) introducing concepts not present in the source text, (c) flawed reasoning, and (d) generating non-critical questions, such as those resembling reading comprehension tasks. parallel to this work, a shared task on critical questions generation has been organized to further stimulate research in this area @xcite . cqs-gen differs from other question-generation tasks @xcite in that its goal is not to produce questions directly answerable from the input text (i.e., reading-comprehension questions). instead, it aims to elicit questions that uncover what remains unsaid, such as hidden premises or implicit connections. moreover, previous work has shown that cqs are useful for fighting misinformation, since they help users identify fallacious reasoning @xcite , and also for predicting the grades of argumentative essays, since they reveal the quality of the arguments @xcite . the potential of using cqs in computational applications has been discussed at length @xcite @xcite . one of the big challenges of generative tasks is evaluation, and cqs-gen is no different. thus, machine translation researchers have long been investigating how to use human-generated references to evaluate new text generations using metrics such as bleurt @xcite , chrf @xcite , and comet @xcite . similarity metrics such as semantic text similarity (sts) @xcite have also been used to compare the new generations to the reference outputs @xcite . finally, recent work has shown the effectiveness of using llms to evaluate various downstream tasks @xcite @xcite , either by using generalpurpose models @xcite , or specialized ones @xcite . in this work, we examine all these evaluation strategies to investigate which one is the most suitable for evaluating the cqs-gen task. ## dataset creation in order to benchmark the capacity of current llms to generate critical questions, a first step is to create a manually annotated dataset. for this purpose, relevant argumentative texts have been collected, and reference questions have been generated and annotated. the rest of the section provides details of each of these steps. ## the task we formalize the task of cqs-gen as a generative task where, when given an input argumentative text, the system has to output exactly 3 critical questions. the 3 questions are then evaluated regarding their usefulness for critically assessing the arguments of the text. for the generation of one useful critical question, the task is scored 1/3, for two 2/3, and if the 3 questions are useful, the task is scored with 1, the maximum punctuation. in figure 2 , an instance of the task with two output examples is provided. for this particular intervention, a question such as "does this argument support socialist policies?" is evaluated as invalid, as it introduces a new concept which is not present in the text: "socialist policies". the question "how does the speaker define "the developed world", and is this a relevant distinction in this context?" gets evaluated as unhelpful, as this definition is not central to the arguments of the text. instead, a similar question such as "how is "sufficient surpluses" defined, and how would one measure it?" gets labeled as useful, as different considerations of what sufficient surpluses are could diminish the strength of the argument. ## experimental settings a major challenge of generative tasks is evaluation. therefore, before benchmarking different models to generate critical questions, we perform extensive research to find the best evaluation methods. for this purpose, we develop two baseline systems, evaluate their output manually, and assess which evaluation method correlates better with human judgments. the best evaluation methods are then applied to establish the first benchmark with 11 state-of-the-art llms. ## results we first present the results on the various automatic evaluation methods tested, and then we report the performance of the 11 llms on the cqs-gen task. ## qualitative analysis while the primary goal of this task is to generate useful critical questions, the diversity of these questions also serves as an important indicator of quality. a broader range of questions may reflect a deeper comprehension of the texts, while a minimal-effort approach characterized by the repeated use of identical or formulaic templates may suggest a superficial understanding and limited capacity for critical engagement. to investigate this issue, we use two metrics, namely, n-gram diversity score and compression ratio diversity score (cr-div) @xcite . the n-gram metric calculates diversity as the ratio of the number of unique ngrams to the total n-grams occurring in the entire set of generated questions. the compression ratio (cr) comes from calculating the ratio of the size of the compressed file to its original size. the cr diversity score can be calculated as the reciprocal of the compression ratio to get a score between 0 and 1, 0 being for identical documents. @xcite shows that this metric is the most reliable one. in the same line, claude-3-5-sonnet also outputs a recursive template regarding evidence, mainly: however, both the 70b qwen model and claude perform substantially better at selecting claims to populate the template than the smaller qwen model, leading to a higher proportion of useful critical questions (see ## conclusion and future work in this work, we have introduced a comprehensive infrastructure for the development and evaluation of systems addressing the task of critical questions generation. first, we constructed a dataset compiled from four different sources, annotated with critical questions categorized as useful, unhelpful, or invalid. second, we explored automatic evaluation methods for this task and found that reference-based approaches yield the most reliable results. finally, we conducted zero-shot evaluations across 11 llms, revealing substantial room for improvement. to support future development, we also provide a public leaderboard for benchmarking cqs-gen systems. although the dataset includes a validation and a test set, our results are based solely on the test set, as we did not pursue parameter or prompt tuning to improve model performance. consequently, exploring methods for improving model performance represents a natural direction for future work. in addition, we aim to investigate the practical utility of generating critical questions for both automated systems and human users. for automated systems, a promising direction is to assess whether incorporating critical questions as intermediate steps in complex reasoning tasks can enhance overall performance. for human users, it could be explored whether exposure to (automatically generated) critical questions could foster improved critical thinking skills.