It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024)

Abstract: Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.

## introduction recent research has aimed to unlock the reasoning capabilities of large language models (llms) @xcite . as part of this effort, researchers have proposed techniques such as chain-of-thought (cot) prompting to help llms verbally reason toward correct answers @xcite . such reasoning can improve the accuracy and interpretability of llm decision-making @xcite . while several works use cot to select correct answers (figure 1 blue), they do not study if cot can identify incorrect answers (figure 1 red). thus, drawing from test-taking strategies for multiplechoice (mc) exams @xcite , we propose a new task to probe llm reasoning: process of elimination (poe) with cot. previous works have performed poe by discarding low-confidence options when selecting correct answers @xcite , question: where do you put your grapes just before checking out? choices: (a) shopping cart (b) mouth correct answer: just before checking out at a grocery store, you typically place your items, including grapes, in the shopping cart. so the correct answer is "shopping cart," which is choice (a) incorrect answer: just before checking out, you would not put grapes in your mouth as that would be consuming them before purchasing. so the incorrect answer is "mouth" which is choice (b) but whether generative llms can directly reason toward incorrect options with cot is unexplored. analyzing poe with cot poses several benefits. first, it can give insights into the consistency of llm reasoning @xcite . for 2-choice mc questions, we can study if the decisions from poe with cot and directly answering the question with cot agree, as both should arrive at the same answer ( §4.2). second, poe with cot can enhance llm interpretability. cot reasoning is often seen as a rationale as to why the model thinks an option is correct, but users may also want to know why the model thinks alternatives are incorrect. third, many applications benefit from poe, such as reaching medical diagnoses via exclusion @xcite , troubleshooting by eliminating root causes @xcite , and ruling out scientific hypotheses using new evidence @xcite . thus, to enable interpretable llms in domains like medicine, customer service, and research, we must know: do llms have the ability to perform poe with cot? to study this question, we independently prompt the gpt-3.5 @xcite , llama-2 @xcite , and falcon @xcite llms to directly select the correct option and eliminate the incorrect one, with and without cot, on 2-choice commonsense @xcite and scientific @xcite reasoning datasets. in this 2choice setup, we can juxtapose llm reasoning abilities in direct answer selection against poe, as ideal llms would always have both strategies agree. we find that poe always underperforms direct answer selection for our llms, but this gap narrows as model size scales, implying that poe with cot may only be attainable for larger llms ( §4.1). further, these two strategies do not reliably arrive at the same answer, revealing an inherent inconsistency in llm reasoning ( §4.2). for more insights, we analyze the errors in the rationales of poe with cot, finding that most stem from reasoning errors and task misalignment ( §4.3), and show how these 2-choice errors propagate when using poe on full mc questions ( §4.4). based on our results, we give suggestions for future work. our contributions are: 1) we introduce poe with cot, a new reasoning task for llms that can benefit interpretability, selfconsistency, and downstream applications. ## problem definition we study selecting correct and incorrect options in mc settings, where we are given a question q and n choices c, exactly one of which is correct. to simplify our discussion, we define two strategies: 1) direct answer (da) strategy: the llm aims to select the correct answer choice c @xmath0. ## experimental setup 3.1 strategy implementation to provide sufficient context for the llms, we use few-shot (10) prompts to implement the strategies, leaving the analysis of 0-shot poe with cot for future work. we construct the following prompts: 1) da base and poe base: these prompts task the llm with performing the da and poe strategies without reasoning. the in-context examples in these prompts follow a similar format to figure 1 , but only the correct/incorrect answer choice ((a) or (b)) follow the "correct/incorrect answer:" labels. to study if cot reasoning improves the accuracy of each strategy ( §4.1), we can compare the accuracy of the base prompts with their cot counterparts (e.g. poe base vs poe cot). further, comparing the outcomes of the da and poe prompts (e.g. da cot vs poe cot) allows us to measure the llms' logical consistency in choosing correct answers and eliminating incorrect options ( §4.2). the cot rationales are written by one ph.d. student and verified by a second (the authors). we also prepend a natural language instruction to the prompt explaining the strategy. more prompting details and examples can be found in appendix a.2. ## conclusion we study cot with process of elimination (poe) and direct answer (da) strategies in 2-choice commonsense and scientific reasoning datasets. we find that poe underperforms da in accuracy across all tested llms, but this gap narrows as model size scales. we also explore the logical inconsistencies between da and poe and categories of errors present in poe rationales. based on our results, we suggest several research directions: using poe with cot as llm reasoning/consistency benchmarks, combining da and poe with cot for calibration and refined rationales, and fine-tuning llms for poe to enhance overall reasoning capabilities.