SPARTA : Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2021)

Abstract: We introduce SPARTA, a novel neural retrieval method that shows great promise in performance, generalization, and interpretability for open-domain question answering. Unlike many neural ranking methods that use dense vector nearest neighbor search, SPARTA learns a sparse representation that can be efficiently implemented as an Inverted Index. The resulting representation enables scalable neural retrieval that does not require expensive approximate vector search and leads to better performance than its dense counterpart. We validated our approaches on 4 open-domain question answering (OpenQA) tasks and 11 retrieval question answering (ReQA) tasks. SPARTA achieves new state-of-the-art results across a variety of open-domain question answering tasks in both English and Chinese datasets, including open SQuAD, CMRC and etc. Analysis also confirms that the proposed method creates human interpretable representation and allows flexible control over the trade-off between performance and efficiency.

## introduction open-domain question answering (openqa) is the task of answering a question based on a knowledge source. one promising approach to solve openqa is machine reading at scale (mrs) @xcite . mrs leverages an information retrieval (ir) system to narrow down to a list of relevant passages and then uses a machine reading comprehension reader to extract the final answer span. this approach, however, is bounded by its pipeline nature since the first stage retriever is not trainable and may return no passage that contains the correct answer. to address this problem, prior work has focused on replacing the first stage retriever with a trainable ranker @xcite @xcite . end-to-end systems have also been proposed to combine passage retrieval and machine reading by directly retrieving answer span @xcite . despite of their differences, the above approaches are all built on top of the dual-encoder architecture, where query and answer are encoded into fixed-size dense vectors, and their relevance score is computed via dot products. approximate nearest neighbor (ann) search is then used to enable realtime retrieval for large dataset @xcite . in this paper, we argue that the dual-encoder structure is far from ideal for open-domain qa retrieval. recent research shows its limitations and suggests the importance of modeling complex queries to answer interactions for strong qa performance. @xcite shows that their best performing system underperforms the state-of-theart due to query-agnostic answer encoding and its over-simplified matching function. @xcite shows the trade-off between performance and speed when moving from expressive crossattention in bert @xcite to simple inner product interaction for dialog response retrieval. therefore, our key research goal is to develop new a method that can simultaneously achieve expressive query to answer interaction and fast inference for ranking. we introduce sparta (sparse transformer matching), a novel neural ranking model. unlike existing work that relies on a sequence-level inner product, sparta uses token-level interaction between every query and answer token pair, leading to superior retrieval performance. concretely, sparta learns sparse answer representations that model the potential interaction between every query term with the answer. the learned sparse answer representation can be efficiently saved in an inverted index, e.g., lucene @xcite , so that one can query a sparta index with almost the same speed as a standard search engine and enjoy the more reliable ranking performance without depending on gpu or ann search. experiments are conducted on two settings: openqa @xcite that requires phraselevel answers and retrieval qa (reqa) that requires sentence-level answers @xcite . our proposed spartaqa system achieves new stateof-the-art results across 15 different domains and 2 languages with significant performance gain, including opensquad, opencmrc and etc. moreover, model analysis shows that sparta exhibits several desirable properties. first sparta shows strong domain generalization ability and achieves the best performance compared to both classic ir method and other learning methods in low-resources domains. second, sparta is simple and efficient and achieves better performance than many more sophisticated methods. lastly, it provides a human-readable representation that is easy to interpret. in short, the contributions of this work include: ## related work the classical approach for openqa depends on knowledge bases (kb)s that are manually or automatically curated, e.g., freebase kb @xcite , nell @xcite etc. semantic parsing is used to understand the query and computes the final answer @xcite . however, kb-based systems are often limited due to incompleteness in the kb and inflexibility to changes in schema @xcite . a more recent approach is to use text data directly as a knowledge base. dr.qa uses a search engine to filter to relevant documents and then applies machine readers to extract the final answer @xcite . it needs two stages because all existing machine readers, for example, bert-based models @xcite , are prohibitively slow (bert only processes a few thousands of words per second with gpu acceleration). many attempts have been made to improve the first-stage retrieval performance @xcite @xcite @xcite . the information retrieval community has shown that word embedding matching do not perform well for ad-hoc document search compared to classic methods @xcite @xcite . to increase the expressiveness of dual encoders, @xcite develops kernel function to learn soft matching score at token-level instead of sequence-level. @xcite proposes poly-encoders to enable more complex interactions between the query and the answer by letting one encoder output multiple vectors instead of one vector. @xcite incorporates entity vectors and multi-hop reasoning to teach systems to answer more complex questions. @xcite augments the dense answer representation with learned n-gram sparse feature from contextualized word embeddings, achieving significant improvement compared to the dense-only baseline. @xcite explores various unsupervised pretraining objectives to improve dual-encoders' qa performance in the low-resources setting. unlike existing work based-on dual-encoders, we focus on learning sparse representation and emphasizing token-level interaction. this is perhaps the most related to the sparse index from den-spi @xcite and @xcite . our approach is different because our proposed model is architecturally simpler and is generative so that it will understand words that not appear in the answer document, whereas the one developed at @xcite ) only models n-grams appear in the document. macavaney et al. (2020) also explores retrieval with sparse representations. our work is different from theirs in that we decide not to model the query order information, which enables the model to do full ranking. section 3.4 shows that our system can be easily deployed via inverted index under modern search engines, such as lucene @xcite . ## openqa experiments we consider an open-domain question answering (openqa) task to evaluate the performance of sparta ranker. following previous work on openqa @xcite @xcite , we experiment with two english datasets: squad @xcite , natural questions (nq) @xcite ; and two chinese datasets: cmrc @xcite , drcd @xcite . for each dataset, we used the version of wikipedia where the data was collected from. preliminary results show that it is crucial to use the right version of wikipedia to reproduce the results from baselines. we compare the results with previous best models. system-wise we follow the 2-stage ranker-reader structure used in @xcite . ranker: we split all documents into sentences. each sentence is treated as a candidate answer a. we keep the surrounding context words of each candidate answer as its context c. we encode at most 512 word piece tokens and truncate the context surrounding the answer sentence with equal window size. for model training, bert-base-uncased is used as the answer encoder for english, and chinesebert-wwm is used for chinese. we reuse the word embedding from corresponding bert model as the term embedding. adam (kingma and ba, 2014) is used as the optimizer for fine-tuning with a learning rate 3e-5. the model is fine-tuned for at most 10k steps and the best model is picked based on validation performance. reader: we deploy a machine reading comprehension (mrc) reader to extract phrase-level answers from the top-k retrieved contexts. for english tasks, we fine-tune on span-bert @xcite . for chinese tasks, we fine-tune on chinesebert-wwm @xcite . two additional proven techniques are used to improve performance. first, we use global normalization @xcite to normalize span scores among multiple passages and make them comparable among each other. second, distant supervision is used. concretely, we first use the ranker to find top-10 passages for all training data from wikipedia corpus. then every mention of the oracle answers in these contexts are treated as training examples. this can ensure the mrc reader to adapt to the ranker and make the training distribution closer to the test distribution @xcite . lastly, evaluation metrics include the standard mrc metric: em and f1-score. ## retrieval qa experiments we also consider retrieval qa (reqa), a sentencelevel question answering task @xcite . the candidate answer set contains every possible sentence from a text corpus and the system is expected to return a ranking of sentences given a query. the original reqa only contains squad and nq. in this study, we extend reqa to 11 different domains adapted from @xcite to evaluate both in-domain performance and out-ofdomain generalization. the details of the 11 reqa domains are in the in-domain scenarios look at domains that have enough training data (see ## conclusion in short, we propose sparta, a novel ranking method, that learns sparse representation for better open-domain qa. experiments show that the proposed framework achieves the state-of-the-art performance for 4 different open-domain qa tasks in 2 languages and 11 retrieval qa tasks. this confirm our hypothesis that token-level interaction is superior to sequence-level interaction for better evidence ranking. analyses also show the advantages of sparse representation, including interpretability, generalization and efficiency. our findings also suggest promising future research directions. the proposed method does not support multi-hop reasoning, an important attribute that enables qa systems to answer more complex questions that require collecting multiple evidence passages. also, current method only uses a bag-ofword features for the query. we expect further gain by incorporating word-order information.