Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering (2022)

Abstract: We propose a simple refactoring of multi-choice question answering (MCQA) tasks as a series of binary classifications. The MCQA task is generally performed by scoring each (question, answer) pair normalized over all the pairs, and then selecting the answer from the pair that yield the highest score. For n answer choices, this is equivalent to an n-class classification setup where only one class (true answer) is correct. We instead show that classifying (question, true answer) as positive instances and (question, false answer) as negative instances is significantly more effective across various models and datasets. We show the efficacy of our proposed approach in different tasks – abductive reasoning, commonsense question answering, science question answering, and sentence completion. Our DeBERTa binary classification model reaches the top or close to the top performance on public leaderboards for these tasks. The source code of the proposed approach is available at https://github.com/declare-lab/TEAM.

## introduction starting with the early text retrieval conference (trec) community-wide evaluations of textual question answering @xcite , all the way to the recent work on multimodal question answering @xcite @xcite and commonsense question answering @xcite , the task has become a staple of the natural language processing research community. one of the major challenges encountered in question answering is the evaluation, which often requires human input to evaluate the textual answers thoroughly. because of this, the alternative that has been proposed is that of multi-choice question answering, where the correct answer is provided together with other incorrect answers. the task is thus transformed into that of answer classification, where a system has to select one answer from the choices provided. while there are drawbacks associated with this evaluation metric, it has been widely adopted because of its benefit of providing a clear evaluation methodology. in this paper, we reformulate the task of multi-choice question answering as a binary classification task and show that this re-framing leads to significant performance improvements on several datasets. importantly, this formulation brings flexibility to the overall question-answering setup, as it reduces the dependence on the up-front availability of multiple candidate answers. using our method -team (two is better than many), candidate answers can be produced and evaluated for correctness on the fly, and thus the answer classification component can be also used in conjunction with more natural settings that use open-ended answer generation @xcite . ## methodology let q be a question for which multiple answer choices @xmath0, . . . , a n } are given. optionally, there is some context c which could be helpful for answering the question. the objective is to select the correct answer a k from the answer set a. for some of the datasets used in the paper, the question q is not provided, and the answer is based only on the context c. for example, swag and hellaswag are two such datasets where the task is to choose the best possible ending for sentence completion, as shown in we discuss how the mcqa task is generally performed using transformer language models in §2.1. we denote this approach as score-based method or score method . we then discuss our proposed binary classification-based method, team in §2.2. ## experimental datasets we experiment with the following datasets: abductive nli @xcite . given two observations o 1 and o 2 (considered as context c), the goal is to select the more plausible intermediate event among hypotheses h 1 and h 2 . we use the sequences {o 1 , h 1 , o 2 } and {o 1 , h 2 , o 2 } as input for both the score and team method. assuming h 1 is the gold answer, we classify {o 1 , h 1 , o 2 } as positive; {o 1 , h 2 , o 2 } as negative. commonsenseqa @xcite or cqa is a dataset for commonsense qa based on knowledge encoded in conceptnet @xcite . given a question, there are five possible choices , among which only one is correct. we do not use any additional knowledge or context for this task. commonsenseqa 2.0 @xcite or cqa2 is a recent challenging qa dataset collected with a model-in-the-loop approach. the dataset contains commonsense questions from various reasoning categories with either yes or no answer. qasc @xcite or question answer- dataset instance cqa question: where on a river can you hold a cup upright to catch water on a sunny day? choice 1: waterfall choice 2: bridge . . . choice 5: mountain qasc question: differential heating of air can be harnessed for what? choice 1: electricity production choice 2: running and lifting choice 3: animal survival . . . choice 8: reducing acid rain swag partial event: on stage, a woman takes a seat at the piano. she ending 1: sits on a bench as her sister plays with the doll. . . . ending 4: nervously sets her fingers on the keys. piqa goal: to separate egg whites from the yolk using a water bottle, you should solution 1: squeeze the water bottle and press it against the yolk. release, which creates suction and lifts the yolk. solution 2: place the water bottle and press it against the yolk. keep pushing, which creates suction and lifts the yolk. cosmosqa @xcite is a qa dataset for commonsense-based reading comprehension. given a question about a paragraph (c), the task is to select the correct answer among four choices. cicero v1, v2 @xcite are datasets for contextual commonsense reasoning in dialogues. given the dialogue and a question about an utterance, the task is to choose the correct answer among multiple choices. we modify the original datasets to use them in a mcqa setup. more details are in the appendix. ## results we use the roberta large @xcite and deberta large @xcite model to benchmark the score and team method across the experimental datasets. we report the accuracy for the validation set in we also report results for other qa systems such as unifiedqa @xcite and uni-corn @xcite for the test set (wherever available) in our main finding is that the team method improves over the score method for most of the datasets except social iqa, physical iqa, and ci-cero v1. we observe this result for both the roberta and deberta models. abductive reasoning: the improvement is consistently large for both validation and test set in the abductive nli (anli) dataset. the problem of intermediate hypothesis selection transforms into a problem of plausible story selection as we use the sequence {o 1 , h, o 2 } as our input. in this formulation, the team method is significantly better than the score method for both roberta and deberta models. science qa: we also observe considerable improvements in the qasc dataset without and with the additional retrieved knowledge. the roberta-team model is more than 7% better in the test set when retrieved knowledge is not used. the difference in performance is around 3% and 4.5% in the validation and test set when the retrieved knowledge is used. for deberta, we observe the most significant improvement in the test results of the qasc-ir setting, where the team method is 3.7% better than the score method. commonsense qa and sentence ending prediction: the team method is also better than the score method for commonsense questionanswering in commonsenseqa and common-senseqa 2.0 across most settings. one notable instance is the 3% superior score of the de-berta team in the commonsenseqa 2.0 validation set. we observe a similar trend in results for sentence-ending prediction in swag and hel-laswag. the improvement in performance for the team method is between 0.85-1.9% in the test set. we also notice improvements in the test set results for reading comprehension qa in cosmosqa. dialogue commonsense reasoning: we observe contrasting results in cicero v1 and v2. the score method outperforms the team method by around 2-3% in cicero v1. however, the team method is better in cicero v2 for both roberta and deberta models. we analyze the results in more detail in §5.1. negative results: the score method outperforms the team method in physical iqa (piqa) and cicero v1. these two datasets contain answer choices that are lexically close together and subtly different from each other (example in ## conclusion in this paper, we introduced a simple binary classification method as an alternative way to address multi-choice question answering (mcqa) tasks. through evaluations on ten different mcqa benchmarks, we showed that this simple method generally exceeds the performance of the score-based method traditionally used in the past. we believe this approach can also be used in the more natural open-ended answer generation setups, thus providing a "bridge" between the mcqa and answer generation frameworks for question answering.