Contextualized Sparse Representations for Real-Time Open-Domain Question Answering (2020)

Abstract: Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (Sparc). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with Sparc, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.

## introduction open-domain question answering (qa) is the task of answering generic factoid questions by looking up a large knowledge source, typically unstructured text corpora such as wikipedia, and finding the answer text segment @xcite . one widely adopted strategy to handle such large corpus is to use an efficient document (or paragraph) retrieval technique to obtain a few relevant documents, and then use an accurate (yet expensive) qa model to read the retrieved documents and find the answer @xcite @xcite . more recently, an alternative approach formulates the task as an end-to-end phrase retrieval problem by encoding and indexing every possible text span in a dense vector offline @xcite . the approach promises a massive speed advantage with 1 code available at https://github.com/jhyuklee/sparc . ## background we focus on open-domain qa on unstructured text where the answer is a text span in a textual corpus (e.g., wikipedia). formally, given a set of k documents x 1 , . . . , x k and a question q, the task is to design a model that obtains the answer â by â = argmax x k i:j f (x k i:j , q), where f is the score model to learn and x k i:j is a phrase consisting of words from the i-th to the j-th word in the k-th document. pipeline-based methods @xcite @xcite typically leverage a document retriever to reduce the number of documents to read, but they suffer from error propagation when wrong documents are retrieved and can be still slow due to the heavy reader model. phrase-indexed open-domain qa as an alternative, @xcite introduce an endto-end, real-time open-domain qa approach to directly encode all phrases in documents agnostic of the question, and then perform similarity search on the encoded phrases. this is feasible by decomposing the scoring function f into two functions, our goal in this paper is to propose a better and learned sparse representation model that can further improve the qa accuracy in the phrase retrieval setup. ## sparse encoding of phrases our sparse model, unlike pre-computed sparse embeddings such as tf-idf, dynamically computes the weight of each n-gram that depends on the context. ## conclusion in this paper, we demonstrate the effectiveness of contextualized sparse representations, sparc, for encoding phrase with rich lexical information in open-domain question answering. we efficiently train our sparse representations by kernelizing the sparse inner product space. experimental results show that our fast open-domain qa model that augments denspi with sparc outperforms previous open-domain qa models, including recent bert-based pipeline models, with two orders of magnitude faster inference time.