Unsupervised Multiple Choices Question Answering: Start Learning from Basic Knowledge (2021)

Abstract: In this paper, we study the possibility of unsupervised Multiple Choices Question Answering (MCQA). From very basic knowledge, the MCQA model knows that some choices have higher probabilities of being correct than others. The information, though very noisy, guides the training of an MCQA model. The proposed method is shown to outperform the baseline approaches on RACE and is even comparable with some supervised learning approaches on MC500.

## introduction question answering (qa) has been widely used for testing reading comprehension. recently, numerous question answering datasets @xcite @xcite @xcite @xcite @xcite @xcite have been proposed. these datasets can be divided into two major categories: extractive question answering (eqa) and multiple choices question answering (mcqa). in eqa, the answer has to be a span of the given reading passage, such as squad @xcite and newsqa @xcite ; while in mcqa, the answer is one of the given choices, such as mctest @xcite and race @xcite . recently, large pretrained language models such as bert @xcite have exceeded human performance in some eqa benchmark corpora, for example, squad @xcite . compared to eqa, mcqa does not restrict the answer to be spans in context. this allowed mcqa can have more challenging questions than eqa, including but not limited to logical reasoning or summarization. the performance gap between bert and human performance is still significant. in this paper, we focus on mcqa. the semi-supervised mcqa model training approach has been proposed @xcite , in which an initial mcqa model is used to answer the unlabelled questions to generate pseudo labeled data. then pseudo labeled data is used to fine-tune the mcqa model to improve the performance. however, the initial mcqa model still needs some labeled examples to train. in this paper, we study the possibility of unsupervised mcqa. instead of starting from an initial mcqa model @xcite , here, the machine starts with some prior knowledge. for example, a choice has a higher probability of being correct if the choice has word overlap with the document and question. with the basic rule, the machine knows that some choices have higher probabilities of being correct than others, and some choices can be ruled out. with these basic rules, an mcqa model can be trained without any labeled mcqa examples. with this approach, we got absolute gains of 4∼9% accuracy compared to the baseline methods on two mcqa benchmark corpora, race and mc500. ## unsupervised mcqa we consider mcqa where we are given a question q, a passage p and a set of choices @xmath0, c 2 , ...c n }, where n is the number of choices, and machine needs to select an answer @xmath1. we propose to address an unsupervised mcqa in a two-stage approach (figure 1 ). first, we pick the candidate set t from choices by fundamental rule from human knowledge (sliding window) or a model trained without mcqa data (eqa model). second, we train a model to pick the final answer from the candidates. ## experiments setup to evaluate the proposed method's effectiveness compared to supervised learning and other approaches that do not require training data, we experiment on two mcqa tasks, race and mctest(mc500). @xcite introduced the race dataset, collected from the english exams for middle and high school chinese students. race consists of near 28000 passages and nearly 100000 questions. specifically, the dataset can be split into two parts: race-m, collected from english examinations designed for middle school students; and race-h, collected from english examinations de-signed for high students. race-h is more difficult than race-m; the length of the passages and the vocabulary size in the race-h are much larger than that of the race-m. @xcite present mctest which requires machines to answer multiple-choice reading comprehension questions about fictional stories. mctest has two variants: mc160, which contains 160 stories, and mc500, which contains 500 stories. moreover, mc500 can be subdivided into mc500-one and mc500-multi. mc500-one refers to the questions that can be answered with one sentence. mc500-multi refers to the questions that need evidence in multiple sentences to answer. the length of each story is approximately 150 to 300 words, and the topic of a story is a wide range. in our experiment, we evaluate our model on mc500 since there are only 280 questions in the mc160, which is not suitable in our setting. ## experiment results race our methods outperform sw and eqa match across all the datasets with absolute gain 4∼9% accuracy, which shows the mcqa model can improve itself from the noisy candidate sets. mml and hard-em outperform highest-only in all cases, which indicates that relying only on the single choice with the highest score is insufficient. the improvement with eqa matching algorithm is more significant than with sw matching algorithm. this implies candidates choosing stage plays a significant role in the performance; more details will be discussed later. ## analysis candidate set & matching methods question types to see how our learning method works with respect to the type of question, we divided the questions in race into six types: why, what, where, when, who, and how. we choose to analyze race because it has more questions than mc 500. figure 2 shows the accuracy of each question types. the results show that the proposed approach does not favor specific types of questions. we found that no matter the candidate set selection methods, the proposed method improved all types of questions, except "where" for eqa and "when" for sw. understanding why some question types do not been improved by unsupervised mcqa in some cases is our future work. ## conclusion in this paper, we proposed an unsupervised mcqa method, which exploits the pseudo labels generated by some basic rules or external non-mcqa datasets. the proposed method significantly outperforms the baseline approaches on race and is even comparable with the supervised learning performance on mc500. we hope this paper sheds light on unsupervised learning in nlp tasks.