A uto EQA : Auto-Encoding Questions for Extractive Question Answering (2021)

Abstract: There has been a significant progress in the field of Extractive Question Answering (EQA) in the recent years. However, most of them are reliant on annotations of answer-spans in the corresponding passages. In this work, we address the problem of EQA when no annotations are present for the answer span, i.e., when the dataset contains only questions and corresponding passages. Our method is based on auto-encoding of the question that performs a question answering task during encoding and a question generation task during decoding. We show that our method performs well in a zero-shot setting and can provide an additional loss to boost performance for EQA.

## introduction extractive question answering (eqa) is the task of finding an answer span to a question from a context paragraph. most of the deep learning models for this task perform well when annotated data is present. scaling such models to new domains often requires creation of new datasets @xcite @xcite . however, collecting labels for these corpora is expensive and time consuming which may involve multiple steps such as article curation, question and answer sourcing. alleviating the annotation efforts for any of these steps is not only of research but also of practical interest. in this work, we address the problem of extracting answer spans to a question from unannotated context paragraph. some works have already been proposed to solve eqa in both semi-supervised and unsupervised setting. unsupervised methods focus on creating a synthetic corpus and further train a supervised model on the synthetic corpus @xcite . in semi-supervised methods the focus is on different pre-training tasks that improve the initialization of the eqa models @xcite it enables the uni-directional language model of the question. we assume a latent distribution over possible answer spans, approximated by candidate phrases. see §2 for details. @xcite . our work can be categorized as the latter with one key difference: to further perform question answering without annotations on answer spans. to validate our approach, we use the pre-trained bert @xcite model using squad @xcite . specifically, our method employs a conditional auto-encoding scheme that reconstructs question given a passage while assuming a latent distribution over the answer phrases. the encoder of our model is a question answering (qa) model that jointly encodes the context and the question to estimate the probability distribution over possible answer spans. this is further given as input along with passage to the decoder which is a question generation (qg) model. we use a shared architecture for both the encoder and the decoder. therefore, our model can be viewed as a self-supervised machine comprehension model that learns from itself. we list our contributions as follows: ## method our model can be characterized as a discrete conditional variational auto-encoder (cvae), where we seek to maximize the ground truth distribution of question given context p θ (q|c) with the assumption that there exist a latent variable answer span. we can then maximize the loglikelihood of p θ (q|c) with this assumption by the evidence lower bound (elbo) (kingma and welling, 2014): ## experiments for eqa experiments, we used the squad v1.1 @xcite dataset and conducted both sentence level and phrase level answer span selection. we trained on paragraph-question pairs without using the labels for answers (i.e., 87, 594 paragraph-question pairs). we maximize the objec- ## related work recently, data augmentation has become a popular way to do unsupervised eqa @xcite @xcite , where synthetic questions are generated either by heuristics or by unsupervised question generation methods. @xcite show that very large-scale language models can generate answers without supervision. while these works have their own benefits, they are different from the problem we intend to address and hence can not be compared directly. for example, @xcite achieves similar performance to ours using millions of artificially created data points for eqa corpora, while we achieve our results by using only 87k training samples suggesting the efficiency of our method when supervision for question, paragraph pairs is provided. ## conclusion in this work, we proposed a novel method for unsupervised answer span selection. we showed that using auto-encoding of question, one can get considerable gains (34.3% em and 53.4% f1 score). methods for unsupervised key phrase extraction can benefit autoeqa in choosing well-informed and dynamic phrases.