From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions (2024)

Abstract: As artificial intelligence reasoning abilities gain prominence, generating reliable benchmarks becomes crucial. The Abstract and Reasoning Corpus (ARC) offers challenging problems yet unsolved by AI. While ARC effectively assesses reasoning, its generation-based evaluation overlooks other assessment aspects. Bloom’s Taxonomy suggests evaluating six cognitive stages: Remember, Understand, Apply, Analyze, Evaluate, and Create. To extend ARC’s focus beyond the Create stage, we developed MC-LARC, a multiple-choice format suitable for assessing stages like Understand and Apply in Large Language Models (LLMs). Our evaluation of ChatGPT4V’s analogical reasoning using MC-LARC confirmed that this format supports LLMs’ reasoning capabilities and facilitates evidence analysis. However, we observed LLMs using shortcuts in MC-LARC tasks. To address this, we propose a self-feedback framework where LLMs identify issues and generate improved options. MC-LARC is available at https://mc-larc.github.io/.

## introduction research on artificial intelligence with reasoning capabilities is attracting attention, leading to the proposal of benchmarks to measure such abilities. the abstraction and reasoning corpus (arc) is one such benchmark designed to evaluate reasoning abilities. each arc task consists of 2-5 examples where both input and output are provided, along with one task where only the input is given. the goal is to infer the rule from the examples and deduce the answer to the task. the input and output grids in arc can range from a minimum 1 × 1 grid to a maximum 30 × 30 grid, with each grid filled with up to 10 different colors. unlike existing reasoning benchmarks, arc's strength lies in its specialization in evaluating reasoning abilities alone by reducing the amount of prior knowledge and data required to solve the tasks. however, arc has limitations in that it is an overly difficult benchmark requiring multiple stages of reasoning to solve. according to bloom's taxonomy @xcite , proposed in traditional educational theory, evaluation consists of the following six stages: remember, understand, apply, analyze, evaluate, and create. in this taxonomy, arc assesses creation, which encompasses all prior levels of cognitive processes, making it difficult to pinpoint which specific stage may be problematic when a solution is not reached. even if the logical reasoning process is correct, the entire response is marked as wrong if there is a slight error in the generated grid. this issue is also found in derived datasets with reduced difficulty, such as mini-arc @xcite and 1d-arc @xcite . although these datasets changed grid sizes or reduced 2d arrays to 1d arrays, it remains difficult to identify which part of the model's reasoning process is flawed when the task is not solved due to the evaluation format that includes creation. therefore, a new evaluation method is needed to identify which step of reasoning is problematic in solving arc. therefore, this paper proposes a modified benchmark called mc-larc to provide an intermediate step in solving arc tasks. mc-larc aims to convert the evaluation format from generation to selection, assessing the areas corresponding to understand and apply in bloom's taxonomy. it converts the dataset into a multiple-choice language format by using large language models (llms) to generate four alternative options based on the correct answer to arc tasks. we conducted experiments to investigate the impact of the transformation into multiple-choice form and found the following two points: 1) the accuracy of llms on arc tasks increased from about 10% to 76%. this modification suggests a narrowing of arc's assessment scope from multiple cognitive processes to primary comprehension. 2) evaluating the extent of the inferential abilities of llms becomes more clearly feasible. however, it was observed that llms used shortcuts while solving mc-larc, finding the correct answer by considering the form or internal context of the choices to eliminate inappropriate options, rather than utilizing reasoning abilities. to address this issue, we introduce a self-feedback framework that leverages llms to improve shortcuts. this method extends beyond previous constraint-based augmentation by incorporating three additional steps: the llm attempts to solve the multiple-choice questions, articulates the problem situations, and then refines the options, thereby autonomously mitigating shortcuts. ## mc-larc: generation to selection we created mc-larc through the following two steps: 1) manually refining the existing larc, and 2) utilizing chatgpt4 to generate wrong options based on larc. refining process the original larc had notable quality issues, as shown in figure 2 . these issues mainly involved 1) inconsistent expressions for the same concept and 2) insufficient details in the provided explanations. for example, the upper part of figure 2 shows different ways of representing the same concepts, causing confusion. this inconsistency could lead to issues when using language models to augment incorrect options, as the arc example imprecise analogy fill in the first, second and last with blue and fill in the other three with yellow. original larc refined answer in mc-larc precise analogy fill in the pixel with blue if the input is symmetrical, and with orange if it is not. flawed analogy refinement refined by experts inconsistent expression example: color grey, pink, yellow, brown pink, yellow, -> brown dark red figure 2: two main issues of larc. (upper) inconsistent terminology: varied expressions for identical concepts (e.g., 'brown' described as 'dark red'). (lower) insufficient problem-solving information: original larc descriptions lack critical details for arc task completion (e.g., symmetry identification). expert revisions fill in these missing details. in addition to the issues highlighted in figure 2 , there were further cases of inconsistency throughout the dataset. these inconsistencies involved not only color but also shape representations and grid manipulation operations. the presence of these multiple issues complicates the process of generating new datasets based on larc, emphasizing the challenges of relying on flawed data sources. to address these issues, we conducted a refining process to enhance quality. this process prioritized ensuring consistency in expressions and rectifying erroneous representations. figure 2 provides an overview of this refining process. ## conclusion to overcome the limitations of the existing arc in measuring inferential reasoning ability, we created a new multiple-choice dataset called mc-larc. as a result, the multiple-choice format allowed for a clearer analysis of logical flow during problemsolving and provided support for the solver's reasoning abilities. however, in an additional control experiment without images, we found that the llm solved problems by finding shortcuts instead of using reasoning abilities. this highlights the regulation needed when using llms to synthesize multiple-choice questions. based on these findings, we introduce a self-feedback framework to address shortcuts. this framework represents our distinctive approach, using llms to generate proper descriptions, thereby mitigating the shortcut problem. these findings have several important implications. firstly, they offer valuable insights into the appropriate methods for evaluating inferential reasoning, demonstrating the potential of using multiple-choice questions for this purpose. secondly, by identifying the constraints to consider when using llms to synthesize multiple-choice questions, this research proposes a framework for the development of more sophisticated and automated high-quality description generators.