Reference-based Weak Supervision for Answer Sentence Selection using Web Data (2021)

Abstract: Answer Sentence Selection (AS2) models are core components of efficient retrieval-based Question Answering (QA) systems. We present the Reference-based Weak Supervision (RWS), a fully automatic large-scale data pipeline that harvests high-quality weakly- supervised answer sentences from Web data, only requiring a question-reference pair as input. We evaluated the quality of the RWS-derived data by training TANDA models, which are the state of the art for AS2. Our results show that the data consistently bolsters TANDA on three different datasets. In particular, we set the new state of the art for AS2 to P@1=90.1%, and MAP=92.9%, on WikiQA. We record similar performance gains of RWS on a much larger dataset named Web-based Question Answering (WQA).

## introduction creating datasets for as2 @xcite , a core task for qa, requires expensive hand-labeling work. we propose the reference-based weak supervision (rws), a fully automatic data pipeline to harvest high quality answers from the web. rws operates in two stages: (i) collecting answer candidates from web documents, and (ii) automatically assigning them correct or incorrect labels. more specifically, we build a large index of more than 100mm web documents from common crawl's crawls. given a question-reference pair, the question is used as a query to retrieve a set of relevant documents from the index. then, we extract sentences from those documents to build a large pool of answer candidates, which are finally scored by an automatic evaluator based on the provided reference. we use the ava approach, which we recently introduced in @xcite for automatic evaluation of as2. we show that rws complements the original data (question/answer pairs) by measuring the improvement over the state-of-the-art as2 models on wikiqa and trec-qa datasets. the experimental results suggest that the weakly supervised data produced by rws adds new supervision capacity to the original dataset, enabling models to advance the state of the art. in a nutshell, our contributions include: (i) a pipeline for processing large-scale data, which generates labeled question-answer pairs using publicly available web data, i.e., common crawl; and (ii) a large automatically labelled dataset derived from the data and labels of asnq @xcite with rws. ## background in this section we provide the background of our work. we first describe as2 task formally, and then introduce tanda, the current state-of-the-art model for as2 @xcite . finally, we present ava employed in our pipeline. ## experiments we study the efficacy of rws by testing its impact on tanda models for as2. we first describe our experimental setup, datasets, and then apply rws to as2-nq. we report the results of tanda when rws's data is used during the transfer stage. ## conclusion we have presented rws a fully automatic data pipeline for as2 that creates a large amount of weakly labeled question-answer pairs from question-reference pairs. this data is showed to benefit as2 models. specifically, we recorded significant performance gains on both popular public benchmarks, wikiqa and trec-qa, and our internal dataset wqa, which is several times larger. in a nutshell, the key motivation of rws is to make use of abundant web data to find more relevant answers for a question. we believe rws can benefit other applications besides as2. we will make our three new datasets, as2-nq, wqa and rws, as well as our index using commoncrawl data available at github.com/alexa/wqa_dataset. we believe this data will enable further research on retrieval-based qa and data creation with weakly supervised techniques.