Critical Questions Generation: Motivation and Challenges (2024)

Abstract: The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it.In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing.Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge.Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton’s argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.

## introduction natural language processing (nlp) applications to deal with misinformation have become a popular line of research in tasks such as fact verification @xcite , evidence retrieval @xcite or counterargument generation @xcite . however, even when deploying generative large language models (llms), most applications face challenges regarding three issues: llms often lack the required up-to-date knowledge for these tasks @xcite , there is not always an agreement on what is the truth @xcite , and llms themselves can produce hallucinations or rely on unfaithful data, generating misinformation of their own making @xcite . yet, instead of requiring the llms to output factual knowledge, could we use them to point at the missing or potentially uninformed claims? in other words, could we use llms to uncover the blind spots in the argumentation? to open this line of research, we ground our work on argumentation theory, which has for centuries been studying dialogical exchanges of information. specifically, we look into argumentation schemes, a set of abstract structures developed by systematically identifying common patterns of argumentation and outlining the defeasibility of these patterns. in these structures, the devices designed to find the blind spots in the arguments are called critical questions. critical questions are the set of inquiries that could be asked in order to judge if an argument is acceptable or fallacious. therefore, these questions are designed to unmask the assumptions held by the premises of the argument and attack its inference. in the theoretical framework developed by @xcite , argumentation schemes are represented as templates depicting the premises, the conclusion, and the critical questions of each scheme. this framework is useful to promote critical thinking, since it allows uncovering fallacies by answering questions. figure 1 shows two examples of argumentation schemes and their corresponding critical questions (cqs). the first of these examples is an argument that links a cause (migration) to an effect (unemployment). therefore, the cqs related to this argument ask about the strength of this relation and the possibility of other causes also having a role in the effect. the second example fits the scheme of practical reasoning. that is, given a goal, the argument defines an action to achieve it. here, the cqs ask about the compatibility of this goal with others, the alternative actions to achiev-figure 1 : arguments from the us2016 dataset @xcite , instantiated using the templates of argumentation schemes and critical questions defined in @xcite . previous work has proved the usefulness of cqs for enhancing fallacy identification @xcite , and for argumentative essays evaluation @xcite . but, to the extent of our knowledge, there has not been any attempt to automate the generation of cqs. in this work, we propose the task of critical questions generation: given an argumentative text, the model is asked to generate the necessary cqs to assess the acceptability of the arguments in the text. in this setting, the argumentative text is the input and the set of cqs is the target output. as in other nlp tasks, such as machine translation or paraphrasing, the model is not required to find new information, but to understand and reformulate the input in a certain way. a crucial requirement to investigate the automatic generation of cqs is to have reference data for experimentation. however, as far as we know, there has not been any attempt to create such a resource. in order to address this shortcoming, in this paper we investigate two methods for creating a dataset for the generation of cqs: (1) using the sets of cq templates defined in @xcite 's theory (from now on, theory-cqs); and to answer these questions, we start by looking at the theoretical sets of cqs and instantiating them using a set of argumentative texts already annotated with argumentation schemes @xcite . as a second step, we prompt two state-of-the-art llms to give us candidate cqs for these same argumentative texts, and we design a procedure to evaluate their relevance towards the texts and their validity as cqs. we then compare the two methods and highlight the main challenges faced by llms when generating cqs. summarizing, the main contributions of this work are: in this work, we observe that questions generated using theory and questions generated using llms are complementary: while theory-cqs are mostly about relations between premises, llm-cqs rather ask about evidences. additionally, llms introduce a new type of questions: those asking about further definition of the terms used in the arguments. regarding the performance of current llms, we observe that models struggle to output relevant cqs and output many non-critical questions. therefore, we conclude that more advanced training and prompting techniques should be used and, to this end, reference data should be created using both the theory and llms' methods. all the data and code in this project has been released. foot_0 ## previous work to contextualise this work, we discuss the relation between argumentation and misinformation, introduce the nature of critical questions, and offer related work on argumentation schemes from a computational point of view. ## data for the purpose of this work we have decided to use a subset of the us2016 @xcite and the moral maze datasets @xcite , which, as explained in the previous section, have already been transcribed and annotated with argumentation schemes. both of these datasets are oral debates, and they are structured as sequences of interventions by different debaters. in order to use these datasets, we have mapped their labels to the argumentation schemes in @xcite . since the labels of both of these datasets are based on walton's work, the mapping has amounted to terminology matching. given the long list of argumentation schemes, we have decided to work with the 18 most frequent schemes. annex a provides the mapping and the distribution of argumentation schemes for each of the two datasets. since both of these datasets have been annotated as iat diagrams, each argumentation scheme label links two or more propositions in the debate, forming an argument. 7 the debates are composed of interventions, which we are going to use as our argumentative texts. each intervention can have many annotated arguments (or none). after preprocessing, 8 we obtain 370 interventions (73 from moral maze and 297 from us2016) of which 117 contain at least one argument (25 from moral maze and 92 from us2016). for the manual analysis of this work, we use 21 of the interventions, chosen to keep the label distribution as similar as possible to the one in the full datasets; 10 of these interventions come from us2016 and 11 from moral maze. the distribution of the 60 arguments contained in these 21 interventions can be found in annex b. ## our method in order to identify the challenges in the task of critical questions generation, there is an urgent need for reference data. to explore how this data should be created, we generate critical questions both using the theory templates and llms. to generate cqs based on walton's theory (theory-cqs), we take each annotated argument and instantiate the cqs associated to that argumentation scheme (red-dotted box at the top of figure 2 ). regarding the generation of cqs with llms (llm-cqs), we prompt two state-of-the-art llms and we evaluate the relevance of the candidate cqs towards the argumentative text (blue-dashed box at the bottom of figure 2 ). we then relate the llm-cqs to the arguments of the text and to the theory-cqs (green box), and assess the validity of the llm-cqs that relate to an argument but do not correspond to any of the existing theory-cqs (such as cq 5 in figure 2 ). in the rest of the section, we describe in detail each of these processes. ## results in order to generate the critical questions we use two open state-of-the-art llms: llama-2-13b and zephyr-13b @xcite . we employ the instruction-tuned chat versions of the models. for zephyr, we use the parameters indicated for their chat version and the chat templates used in training. for llama-2, we use the chat version released in july 2023. with the two prompts, we obtain 495 llm-generated candidate cqs (llm-cqs). we now report the results of each of the evaluation steps described in section 4.2, to later compare the llm-cqs to the theory-cqs, showing the differences between the questions obtained through each of these approaches. ## concluding remarks in this work we have introduced and motivated the task of critical questions generation. moreover, we have studied how to generate valid critical questions with two goals in mind: (i) designing a procedure to obtain reference data, and (ii) discovering the main difficulties that state-of-the-art llms face when generating valid critical questions. regarding the difficulties of the task, we have found that current llms struggle to generate cqs mt: "claire's absolutely right about that. but then the problem is that that form of capitalism wasn't generating sufficient surpluses. and so therefore where did the money flow. it didn't flow into those industrial activities, because in the developed world that wasn't making enough money." (a) intervention -how strong is the generalisation that if that form of capitalism was not making enough money in the developed world then the money did not flow into those industrial activities? -are there other factors in this particular case that could have interfered with the event of 'the money did not flow into those industrial activities'? -how strong is the generalisation that if that form of capitalism wasn't generating sufficient surpluses then the money did not flow into industrial activities? (b) theory-cqs -how is 'sufficient surpluses' defined, and how would one measure it? -is mt implying that current forms of capitalism are more successful at generating profits and surpluses than the one being discussed? if yes, why? -what evidence is there to support the claim that the form of capitalism being used in the developed world was not generating sufficient surpluses? -are there any alternative explanations for why the money did not flow into industrial activities? (c) llm-cqs strictly related to the text. on the one hand, they tend to output cqs including new concepts not present in the arguments. on the other hand, they sometimes opt for generating unfiltered lists of very general cqs, with no regard to the given argumentative text. reasoning is still an issue for these models, and they sometimes struggle to understand what claims are actually held by the given text. finally, while 62% of the llm-generated cqs did not have any of these three issues (308 out of 495), only 28% of the cqs initially generated by llms were found to be valid in relation to one of the arguments (137 out of 495), showing that there is a big margin for improvement. in relation to the goal of creating a reference dataset, we have shown that the existing theoretical sets of critical questions do not account for all the possible valid critical questions. in this sense, our results show that only 25% of the valid llm-cqs had been included in the theoretical sets (52 out of 207). for this reason, we propose using both theory-cqs and llm-cqs to build the reference data for this task. furthermore, we have also observed that the type of questions generated by llms differs from the ones created by theory, with the llms approach generating many questions related to evidence, consequences and definitions. this suggests that the two approaches (theory and llms) are complementary. while this work has been a first step towards the task of critical questions generation, our end goal of automatically generating valid cqs is far from solved. in future work, we will create a larger reference dataset including both theory and llm-cqs to facilitate research on automatic cqs generation. finally, it should be noted that we have not paid any attention to llm-generated questions that did not match any of the annotated arguments. however, as some arguments might be missing from the annotation (either because they were not in our selected 18 argumentation schemes or because the annotators missed them), some of these questions might be valid cqs. this shows that our work relies heavily on already annotated data with argumentation schemes. and, while the datasets used are reliable @xcite , there is not a lot of quality data annotated with argumentation schemes, which poses a limitation on how much reference data can be created. as far as we are aware, the only data available is the one detailed in