M al A lgo QA : Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education (2024)

Abstract: This paper introduces MalAlgoQA, a novel dataset designed to evaluate the counterfactual reasoning capabilities of Large Language Models (LLMs) through a pedagogical approach. The dataset comprises mathematics and reading comprehension questions, each accompanied by four answer choices and their corresponding rationales. At the heart of MalAlgoQA are “malgorithms” - rationales behind incorrect answer choices that represent flawed yet logically coherent reasoning paths. These malgorithms serve as counterfactual scenarios, allowing us to assess an LLM’s ability to identify and analyze flawed reasoning patterns. We propose the Malgorithm Identification task, where LLMs are assessed based on their ability to identify corresponding malgorithm given an incorrect answer choice. To evaluate the model performance, we introduce two metrics: Algorithm Identification Accuracy (AIA) for correct answer rationale identification, and Malgorithm Identification Accuracy (MIA) for incorrect answer rationale identification. Our experiments reveal that state-of-the-art LLMs exhibit significant performance drops in MIA compared to AIA, highlighting the challenges in counterfactual reasoning.Surprisingly, we find that the chain-of-thought prompting technique not only fails to consistently enhance MIA but can sometimes lead to underperformance compared to simple prompting. These findings have important implications for developing LLMs with improved counterfactual reasoning, particularly relevant for AI-powered tutoring systems, where identifying and addressing student misconceptions is essential. MalAlgoQA dataset is available here.

## introduction evaluating the counterfactual reasoning capabilities of large language models (llms) -their * equal contribution. ability to consider "what if" scenarios and infer causes based on hypothetical situations -remains a significant challenge despite their impressive performance in various natural language tasks @xcite @xcite . we introduce a novel method to evaluate these capabilities, drawing inspiration from educational assessment techniques. specifically, we leverage the practice of distractor choice generation, which involves creating plausible but incorrect answer options by envisioning hypothetical scenarios and logical yet flawed reasoning paths. this approach mimics how educators anticipate and model potential student misconceptions, engaging in counterfactual thinking about alternative cognitive processes that could lead to specific errors. building on this foundation, we present malal-goqa, a dataset designed to challenge llms' ability to reason about hypothetical situations and identify flawed logical pathways. malalgoqa consists of a diverse set of 807 mathematics and 290 reading comprehension questions spanning various grade levels (3-11), content classifications (e.g., algebra, geometry, number & operations), and depth of knowledge levels (dok) (1-3) @xcite . each question is accompanied by a set of answer choices and their associated rationales. central to our approach is the concept of "malgorithms" -a term we coin to describe the flawed reasoning paths that lead to incorrect answers (name is inspired by the concept of "malrules" proposed by @xcite ). these malgorithms represent deterministic yet erroneous thought processes, mirroring the way students might logically follow a series of steps based on a misunderstanding or misapplication of concepts. we propose the malgorithm identification task to assess llms' ability to reason about both correct and incorrect answer rationales. in this task, given a question and a specific answer choice, the model must identify the underlying rationale ## malalgoqa dataset we introduce malalgoqa, a real-world dataset consisting of multiple-choice questions in mathematics and reading comprehension for students in grade 3 through 11. the dataset is designed to evaluate the counterfactual reasoning abilities of llms through the task of analyzing the rationales and reasons behind incorrect answer choices, a process we refer to as "malgorithm identification." the malalgoqa dataset provides valuable resources to assess the counterfactual reasoning capabilities of llms through a pedagogical approach. ## methodology in this section, we present the task formulation for malgorithm identification. ## analysis: educational dimensions performance across grade levels performance across dok figure 1 presents a comprehensive overview of performance across dok levels for various grade using mia score from gpt 4o with cot prompting strategy. the pattern indicates that as the dok level and grade increases, mia performance tends to decline, especially highlighted by the comparison between the top-left cell (dark green) and bottom-right cell (dark red). similarly, in the reading dataset, performance at dok level iii drops to 33.33 for grade 7. this trend further emphasizes the inherent difficulty in evaluating the malgorithms of questions that require deeper understanding and analytical thinking. ## implications for ai in education the findings from our study on malgorithm identification have significant implications for the development and application of ai in educational contexts: limitations in understanding student misconceptions: the substantial performance gap between algorithm identification accuracy (aia) and malgorithm identification accuracy (mia) suggests that current llms are far more adept at recognizing correct reasoning than identifying flawed logic. for gpt-4o, we observed a drop from 95.65% aia to 66.10% mia on the math dataset. this disparity indicates that ai tutoring systems based on these models may excel at confirming correct answers but struggle to diagnose and address student misconceptions effectively. this limitation could lead to missed opportunities for targeted intervention and personalized learning. ## related work counterfactual reasoning, a critical cognitive process for understanding causality, has gained significant attention in machine learning (ml) and nlp research. pearl's seminal work on causal inference @xcite has been widely applied to address challenges in fairness @xcite , interpretability @xcite , and robustness @xcite in ml. in nlp, counterfactual reasoning is important for tasks requiring comprehension of hypothetical scenarios. however, most existing nlp models primarily focus on predictive tasks rather than hypothetical reasoning @xcite . recent work has begun to address this gap developing specialized datasets and models for counterfactual reasoning @xcite . several datasets have been created to evaluate different aspects of reasoning, including counterfactual reasoning such as copa dataset @xcite , counterfactual story rewriting task @xcite , and cqa dataset @xcite . these efforts are complemented by development of new modeling approaches, such as causal lms @xcite , counterfactual data augmentation @xcite , and generating counterfactual explanations @xcite . despite these advancements, challenges remain such as handling multi-step causal chains and generalizing across diverse domains @xcite @xcite . ## conclusion and future work this paper introduces malalgoqa, a novel dataset designed to evaluate the counterfactual reasoning capabilities of llms through a pedagogical lens. by proposing the malgorithm identification task and its associated metrics, we provide a new framework for assessing llms' ability to recognize and analyze flawed reasoning patterns. our findings reveal significant challenges in counterfactual reasoning for state-of-the-art llms. these results highlight the need for further research into enhancing llms' counterfactual reasoning abilities, particularly for educational applications where understanding student misconceptions is crucial. future work will focus on expanding the dataset, exploring novel training methodologies, and investigating architectural modifications to develop more cognitively-aligned language models capable of robust counterfactual reasoning.