Winnowing Knowledge for Multi-choice Question Answering (2021)

Abstract: We tackle multi-choice question answering. Acquiring related commonsense knowledge to the question and options facilitates the recognition of the correct answer. However, the current reasoning models suffer from the noises in the retrieved knowledge. In this paper, we propose a novel encoding method which is able to conduct interception and soft filtering. This contributes to the harvesting and absorption of representative information with less interference from noises. We experiment on CommonsenseQA. Experimental results illustrate that our method yields substantial and consistent improvements compared to the strong Bert, RoBERTa and Albert-based baselines.

## introduction multi-choice question answering (mqa for short) is required to select an answer from a set of candidate options when given a question @xcite . the task is slightly different from multichoice reading comprehension which provides the passage containing background knowledge for reasoning @xcite . frankly, due to the lack of commonsense knowledge, mqa is more challenging. for example, it appears to be difficult for mqa to determine the true answer in the following case, where the commonsense knowledge regarding "island country" deterministically contributes to reasoning, though such knowledge is not offered in any form by default: we follow the previous work to perform mqa using external knowledge bases. information retrieval is utilized for knowledge acquisition as usual. the difference is that we intend to enhance the joint encoding of question, option and knowledge by soft filtering and interception. the filter is used to shield the encoder from the negative influence of the mistakenly-retrieved irrelevant or unrepresentative knowledge (called noise hereafter). it is motivated by our findings that purposefully retrieving a larger number of knowledge items actually results in performance degradation. as illustrated in figure 1 , the performance curve of the retrieval-based mqa model (green curve) shows a trend of fluctuating downward when the number of the adopted highly-ranked search results is increased. it is most likely caused by the noises that sneak into the list of retrieval knowledge items. instead of thoroughly filtering noises out from the retrieval knowledge, we perform soft filtering which retains noises but merely assigns negligible attention weights to them. on the basis, we develop an interceptor to "eavesdrop" on the encoding channel of knowledge items, and salvage the recyclable latent information hidden in them. it is motivated by the fact that part of the content of a certain less-relevant knowledge item is probably informative. see the knowledge item in (2), in which the constituent "a large island" is informative and recyclable (as it even bridges the key words "great britain" in the relevant knowledge and "island country" in the question). to recycle available evidence in retrieval knowledge, the interceptor conducts information fusion among them, conditioned on the assignment of interactive attention to them. we implement the interceptor and soft filter by self-attention network and attention pooling layer, which are collectively referred to as "headhunter". we couple a certain pre-trained model with headhunter for encoding, and deploy them along with elasticsearch in the commonly-used two-stage mqa architecture. we experiment over the com-monsenseqa dataset @xcite . experimental results show that headhunter yields significant performance gains all along when coupled with different pre-trained models, including bert @xcite , roberta @xcite and albert @xcite . besides, the case of combining albert and headhunter achieves better performance than most state-of-the-art models, and it is ranked second on the commonsenseqa leaderboard for track 1. the developed results show that the performance advantages can be safely attributed to the constraint on the use of less-relevant knowledge in headhunter. to some extent, it successfully avoids the severe performance degradation when a larger set of qualified and less-qualified commonsense knowledge is taken (see the red curve in figure 1 ). ## conclusion we develop a vest-pocket model to squeeze reliable information out of commonsense knowledge as completely as possible. it is proven beneficial to 3 we run the models on an nvidia tesla v100 sxm2 16gb gpu (volta microarchitecture). mqa performance when cooperating with bert, roberta and albert-based encoders. error analysis demonstrates that a critical bottleneck lies in the disambiguation towards indistinguishable options. in the future, we will study the dictionary-based disambiguation approach, detecting and representing the most distinct aspects of words in terms of definitions. moreover, a multi-task learning architecture will be developed, where knowledge acquisition, word sense disambiguation and mqa share the encoding channels of both general and distinctive word senses (named entities are not included). appendix.