Instructing Large Language Models to Identify and Ignore Irrelevant Conditions (2024)

Abstract: Math word problem (MWP) solving requires generating a reasoning path based on a given problem description that often contains irrelevant conditions.Existing chain-of-thought (CoT) prompting methods elicited multi-step reasoning abilities of large language models (LLMs) to solve MWPs.However, they were seriously confused by the irrelevant conditions, resulting in low accuracy.In this paper, we propose a novel approach named I ^3 C that instructs LLMs to identify and ignore irrelevant conditions.It identifies a set of irrelevant condition candidates that have a weak semantic relevance with the question.Then it prompts LLMs to verify the irrelevant conditions.Lastly it instructs the LLMs with the verification on relevant and irrelevant conditions to avoid confusion and improve reasoning paths.Moreover, we propose to select (problem, reasoning paths) pairs as demonstrations to enhance I ^3 C with few-shot reasoning. We develop I ^3 C-Select that selects the most confusing problems based on the semantic relevance measurement.We conduct extensive experiments on eight MWP datasets.I ^3 C can be combined with any CoT prompting methods to improve the performance of solving MWPs.Notably, with GPT-3.5-Turbo and I ^3 C-Select, we achieve an accuracy of 96.0 and 94.1 on GSM-IC2-1K and GSM-ICM-1K, respectively, significantly outperforming the state-of-the-art few-shot prompting method Complex-CoT by +11.7 and +11.1 .Our implementation is made publicly available at https://wzy6642.github.io/I3C.github.io/.

## introduction math word problem (mwp) solving is a task of developing algorithms to generate a reasoning path towards an unknown quantity based on a problem description. this task is challenging as it requires mathematical understanding and multi-step reasoning abilities. chain-of-thought (cot) prompting * corresponding author methods were able to guide large language models (llms) to perform complex multi-step reasoning @xcite . adding demonstrations created manually @xcite or retrieved from a large training set @xcite in cot prompts enabled few-shot in-context learning and improved accuracy. however, @xcite found that existing cot prompting methods could be seriously confused by irrelevant conditions which are specifications or data presented in a problem that are unrelated to the solution @xcite . for example, as shown in figure 1a , the third condition "the height of mary is 5 feet." was irrelevant to the final question and misled the reasoning and prediction. shi et al. added a plain instruction "feel free to ignore irrelevant conditions in the problem description." in the prompts, but the llms could not effectively ignore them in the problem solving process because they were not identified or specified in the instruction. improving the reasoning on mwps that have irrelevant conditions is non-trivial. self-consistency @xcite was proposed to repeatedly solve a problem multiple times (e.g., 10 times) and employ a majority vote strategy to determine the most consistent answer as the final answer. however, it was computationally expensive and still confused by the irrelevant conditions. moreover, the demonstrations would have to be re-designed to obtain the few-shot learning ability of identifying and ignoring the irrelevance, compared to those in @xcite . in this paper, we propose a novel approach, i 3 c, to instruct llms to explicitly identify and ignore irrelevant conditions in the mathematical reasoning process. it creates effective instructions that can be added to any cot prompts to improve their generated reasoning paths. unlike self-consistency, i 3 c does not prompt llms multiple times. its advanced variant, i 3 c-select, uses the most confusing problems and their generated reasoning paths as 𝑐 ! steve is 5'6". ## conclusion in this study, we introduce a plug-and-play module, i 3 c, which can be added to any cot prompting methods to enhance llms' ability to explicitly identify and ignore irrelevant conditions in the mathematical problem-solving process. moreover, we propose a novel few-shot prompting method, i 3 c-select, which selects the most confusing problems and their corresponding reasoning paths as demonstrations. extensive experiments on eight math word problem datasets demonstrate the effectiveness and efficiency of our proposed method.