Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation (2021)

Abstract: Early fusion models with cross-attention have shown better-than-human performance on some question answer benchmarks, while it is a poor fit for retrieval since it prevents pre-computation of the answer representations. We present a supervised data mining method using an accurate early fusion model to improve the training of an efficient late fusion retrieval model. We first train an accurate classification model with cross-attention between questions and answers. The cross-attention model is then used to annotate additional passages in order to generate weighted training examples for a neural retrieval model. The resulting retrieval model with additional data significantly outperforms retrieval models directly trained with gold annotations on Precision at N (P@N) and Mean Reciprocal Rank (MRR).

## introduction open domain question answering (qa) involves finding answers to questions from an open corpus @xcite @xcite . the task has led to a growing interest in scalable end-to-end retrieval systems for question answering. when qa is formulated as a reading comprehension task, cross-attention models like bert @xcite have achieved better-than-human performance on benchmarks such as the stanford question answering dataset (squad) @xcite . cross-attention models are especially well suited for problems involving comparisons between paired textual inputs, as they provide early fusion of fine-grained information within the pair. this encourages careful comparison and integration of details across and within the two texts. however, early fusion across questions and answers is a poor fit for retrieval, since it prevents precomputation of the answer representations. rather, neural retrieval models independently compute embeddings for questions and answers, typically using dual encoders for fast scalable search @xcite @xcite . using dual encoders results in late fusion within a shared embedding space. for machine reading, early fusion using crossattention introduces an inductive bias to compare fine grained text spans within questions and answers. this inductive bias is missing from the single dot-product scoring operation of dual encoder retrieval models. thus, late fusion is expected to require more training data to learn the necessary representations for fine grained comparisons. to support learning improved representations for retrieval, we explore a supervised data augmentation approach leveraging a complex classification model with cross-attention between questionanswer pairs. given gold question passage pairs, we first train a cross-attention classification model as the supervisor. then any collection of questions can be used to mine potential question passage pairs under the supervision of the cross-attention model. the retrieval model training benefits from additional training pairs annotated with the graded predictions from the cross-attention model augmenting the existing gold data. experiments on multireqa-squad and multireqa-nq establish significant improvements on precision at n (p@n) and mean reciprocal rank (mrr). the supervised mining approach is closely connected to the recently studied hard negative mining for neural retrieval models @xcite . the key differences is that the proposed approach finds the positive training examples, while the negative mining approaches find the negative examples for training. the two approaches are complementary and can be combined. ## neural passage retrieval for open domain question answering open domain question answering systems usually follow a two-step approach: first retrieve question relevant passages, and then scan the returned text to identify the answer span using a reading comprehension model @xcite @xcite . prior work has focused on the answer span annotation task and has even achieved super human performance on some datasets. however, the evaluations implicitly assume the trivial availability of passages for each question that are likely to contain the correct answer. while the retrieval task can be approached using traditional keyword based retrieval methods such as bm25, there is a growing interest in developing more sophisticated neural retrieval methods @xcite @xcite . ## methodology in this section we describe the proposed approach using a neural retrieval model augmented with su-1 https://github.com/ google-research-datasets/multireqa ## evaluation in this section we evaluate the proposed approach using the multireqa evaluation splits for nq and squad. models are assessed using precision at n (p@n) and mean reciprocal rank (mrr). following the reqa setup @xcite , we report p@n for @xmath0. p@n evaluates whether the true answer sentence appears in the top-n ranked candidates. mrr is calculated as @xmath1, where n is the total number of questions, and rank i is the rank of the first correct answer for the ith question. ## conclusion in this paper, we propose a novel approach for making use of an early fusion classification model to improve late fusion retrieval models. the early fusion model is used for data mining to augment the training set for the late fusion model. the proposed approach mines 53% (56,148) and 12% @xcite more examples for multirqa-nq and multirqa-squad, respectively. compared to the models directly trained with gold annotations, the resulting retrieval models improve +8.6% and +1.0% p@1 on nq and squad respectively. the current pipeline assumes there exists annotated indomain question answer pairs to train the crossattention model. with a strong general purpose cross-attention model, our method could be modified to train in-domain retrieval models without gold data. we leave this to the future work.