POE : Process of Elimination for Multiple Choice Reasoning (2023)

Abstract: Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.

## introduction how often have i said to you that when you have eliminated the impossible whatever remains, however improbable, must be the truth? @xcite in natural language processing, many reasoning tasks are multiple choice-based, in which a model chooses the best option from several options, given a question. current lms exhibit remarkable performance on diverse reasoning tasks, like commonsense reasoning @xcite , logical reasoning @xcite , and arithmetic reasoning @xcite . there are two types of in-context learning methods for multiple choice reasoning: scoring and prompting. given the question, scoring methods score each option, and select the highest-scored option. one common score is language modeling likelihood (p (y i |x), @xcite , and there are also many alternatives @xcite @xcite @xcite @xcite . tailored for powerful llms @xcite , prompting methods @xcite @xcite @xcite append all options to the question, and prompt the model to generate a sequence, then extract the option from the sequence. one limitation of these two types of approaches is that they both treat each option equally, i.e., they either consider each option independently, or consider all options jointly. on the contrary, when humans solve a multiple choice reasoning task, they often eliminate some wrong options, then choose from the remaining ones. this so-called process of elimination is widely used in college exams and can be stronger than usual gold style learning @xcite . therefore, we assume that language models can similarly benefit from this elimination process, i.e., eliminating wrong options and choosing the best option are two types of reasoning that can be disentangled into two steps. to this end, we present the process of elimination (poe), a two-step scoring method for multiple choice reasoning, as shown in figure 1 . in the first step, poe scores each option, then eliminates some wrong options based on their scores. in the second step, it masks these wrong options, then chooses the best one from the remaining options. we conduct experiments on 8 reasoning tasks that cover diverse domains. poe achieves the best zero-shot performance on most tasks. a following analysis shows that it favors logical reasoning tasks. we also measure the effect of masks, and find our method applicable to few-shot settings and compatible with llms like chatgpt @xcite . our contributions are twofold: (1) we present poe, a two-step scoring method for multiple choice reasoning; (2) through comprehensive experiments and analysis, we demonstrate the effectiveness and generalizability of poe. scoring uses an lm to compute a score for each option y i , and chooses the highest-scored option: a commonly-used score is language modeling likelihood (p (y i |x), @xcite , more recent scores include average log probability @xcite , calibrated log probability @xcite , and channel @xcite . as shown in equation 1, most scoring methods consider each option in isolation, except multiple choice prompting @xcite . in contrast, prompting methods provide all the options in the input @xcite @xcite . the model then generates raw output from the input. finally, these methods extract the option from the raw output: our method is a two-step scoring method that considers all options but treats them differently. we also implement a prompting-based variation of poe for llm (section 5). the first step of poe is elimination, in which it eliminates some wrong options. poe starts by scoring each option. then, unlike common scoring methods that choose the highest-scored option, it uses the scores to eliminate some options with low scores. in particular, poe computes the average score of all options, and eliminates options whose score is below average, i.e., y wrong : the intuition behind this elimination strategy ("below average") is that the scores of wrong options deviate from others, and are thus easy to identify, which we verify in appendix b.2. we compare other elimination strategies in section 5.2. the second step of poe is prediction, which chooses the best answer that is not in y wrong . specifically, poe computes binary masks m i for options: for these options, poe enforces the masks. 2 in particular (shown in figure 1 ), it uses a template t to first wrap the question with all options and their symbols like "a", then append an instruction to the question which asks the model to neglect masked options, and finally replace eliminated options with a special text sequence "[mask]": then, poe scores each option, and chooses the highest-scored option, during which the scores of eliminated options are set to negative infinity: model. we use flan-t5-xl (3b) @xcite in both steps of poe, because it is an economical and performant instruction-tuned model. in section 5, we also apply poe to llms like chatgpt @xcite . 3 methods. we consider 5 scoring baselines: language modeling (lm, baseline in @xcite , average language modeling (avg, @xcite , calibration @xcite , channel @xcite , and multiple choice prompting (mcp, @xcite . for poe, we use mcp in both steps. we discuss other possible scoring methods and elimination strategies for poe in section 5.2, and present input and output samples for all methods in appendix c. foot_3 ## results in ## conclusion we present poe, a two-step scoring method for multiple choice reasoning. poe eliminates some wrong options in the first step, and chooses from the remaining options in the second step. poe performs well on reasoning (especially logical reasoning) tasks in the zero-shot setting, and also works with llms or in the few-shot setting. in the future, we will improve the generalizability of our method, and use fine-tuning to better enforce the masks.