Context-Aware Answer Extraction in Question Answering (2020)

Abstract: Extractive QA models have shown very promising performance in predicting the correct answer to a question for a given passage. However, they sometimes result in predicting the correct answer text but in a context irrelevant to the given question. This discrepancy becomes especially important as the number of occurrences of the answer text in a passage increases. To resolve this issue, we propose BLANC (BLock AttentioN for Context prediction) based on two main ideas: context prediction as an auxiliary task in multi-task learning manner, and a block attention method that learns the context prediction task. With experiments on reading comprehension, we show that BLANC outperforms the state-of-the-art QA models, and the performance gap increases as the number of answer text occurrences increases. We also conduct an experiment of training the models using SQuAD and predicting the supporting facts on HotpotQA and show that BLANC outperforms all baseline models in this zero-shot setting.

## introduction question answering tasks require a high level of reading comprehension ability, which in turn requires a high level of general language understanding. this is why the question answering (qa) tasks are often used to evaluate language models designed to be used in various language understanding tasks. recent advances in contextual language models brought on by attention @xcite @xcite and transformers @xcite have led to significant improvements in qa, and these improvements show that better modeling of contextual meanings of words plays a key role in qa. while these models are designed to select answer-spans in the relevant contexts from given passages, they sometimes result in predicting the this passage has multiple spans that are matched with the answer text. the first occurrence of "prefrontal cortex" is the only answer-span within the context of the question. tackling this problem is very important in more realistic datasets such as naturalquestions @xcite , where the majority of questions have more than one occurrence of the answer text in the passage. this is in contrast with the squad dataset, where most questions have a single occurrence of the answer. these details of the squad @xcite , newsqa, and naturalquestions datasets @xcite are shown in figure 3 . to address this issue, we define context prediction as an auxiliary task and propose a block attention method, which we call blanc (block attention for context prediction) that explicitly forces the qa model to predict the context. we design the context prediction task to predict softlabels which are generated from given answerspans. the block attention method effectively calculates the probability of each word in a passage being included in the context with negligible extra parameters and inference time. we provide the implementation of blanc publicly available foot_0 . adding context prediction and block attention enhances blanc to correctly identify context related to a given question. we conduct two types of experiments to verify the context differentiation performance of blanc: extractive qa task, and zero-shot supporting facts prediction. in the extractive qa task, we show that blanc significantly increases the overall reading comprehension performance, and we verify the performance gain increases as the number of answer texts in a passage increases. we verify blanc's contextaware performance in terms of generalizability in the zero-shot supporting facts prediction task. we train blanc and baseline models on squad1.1. and perform zero-shot supporting facts (supporting sentences in passages) prediction experiment on hotpotqa dataset @xcite . the results show that the context prediction performance that the model has learned from one dataset is generalizable to predicting the context of an answer to a question in another dataset. contributions in this paper are as follows: ## related work evidence in the form of documents, paragraphs, and sentences, has been shown to be necessary and effective in predicting the answers in open-domain qa @xcite @xcite and multi-hop qa @xcite @xcite . one problem of identifying evidence in answering questions is the expensive cost in labeling the evidence. self-labeling with simple heuristics can be a solution to this problem, as shown in @xcite ; @xcite . self-training is another solution, as presented in @xcite . in this paper, we propose self-generating soft-labeling method to indicate support words of answer texts, and train blanc with the soft-labels. related but different from our work, swayamdipta et al. (2018) and min et al. (2019a ) predict the answer-span when only the answer texts are provided and the ground truth answer-spans are not. @xcite designs a model that benefits from aggregating information from multiple mentions of the answer text in predicting the final answer. @xcite approach the problem of the lack of ground truth answer-spans with latent modeling of candidate spans. both of these papers tackle the problem of identifying the correct answer among multiple mentions of the answer text in datasets without annotations of the correct answer-spans. our work solves a different problem from the above-mentioned papers in that the golden answer-spans are provided. ## model we propose blanc based on two novel ideas: softlabeling method for the context prediction and a block attention method that predicts the soft-labels. two important functionalities of blanc are 1) calculating the probability that a word in a passage belongs to the context, which is in latent, and 2) enabling the probability to reflect spatial locality between adjacent words. we provide an overall illustration of blanc in figure 4 . ## experimental setup we validate the efficacy of blanc on two types of tasks: extractive qa and zero-shot supporting fact prediction. in the extractive qa, we evaluate the overall reading comprehension performance with three qa datasets, and we further analyze the ability of blanc to discern relevant contexts on passages with multiple answer texts. in zero-shot supporting facts prediction, we train qa models on squad @xcite and predict supporting facts (supporting sentences) of answers in hotpotqa @xcite . due to our experimental computing resource limitation, we compare blanc to baseline models trained in slightly modified hyperparameter settings instead of the results from their original papers. ## results & discussion we now present the results for the experiments described in the previous section. we describe the overall reading comprehension performance, highlighting the increased gain for passages with multiple mentions of the answer text. we show that blanc outperforms other models for zero-shot supporting fact prediction. we also demonstrate the importance of the context prediction loss and the negligible extra parameter and inference time. ## conclusion in this paper, we showed the importance of predicting an answer with the correct context of a given question. we proposed blanc with two novel ideas: context word prediction task and a block attention method that identifies an answer within the context of a given question. the context words prediction task labels latent context words with the labeled answer-span and is used in a multi-task learning manner. block attention models the latent context words with negligible extra parameters and training/inference time. we showed that blanc increases reading comprehension performance, and we verify that the performance gain increases for complex examples (i.e., when the answer occurs two or more times in the passage). also, we showed the generalizability of blanc and its contextaware performance with the zero-shot supporting fact prediction task on the hotpotqa dataset. zhilin yang, peng qi, saizheng zhang, yoshua bengio, william cohen, ruslan salakhutdinov, and christopher d manning. 2018. hotpotqa: a dataset for diverse, explainable multi-hop question answering. in emnlp.