Simple Entity-Centric Questions Challenge Dense Retrievers (2021)

Abstract: Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples. However, in this paper, we demonstrate current dense models are not yet the holy grail of retrieval. We first construct EntityQuestions, a set of simple, entity-rich questions based on facts from Wikidata (e.g., “Where was Arve Furset born?”), and observe that dense retrievers drastically under-perform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions towards addressing this critical problem. First, we demonstrate that data augmentation is unable to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using specialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.

## introduction recent dense passage retrievers outperform traditional sparse retrieval methods like tf-idf and bm25 @xcite ) by a large margin on popular question answering datasets @xcite @xcite . these dense models are trained using supervised datasets and the dense passage retriever (dpr) model @xcite demonstrates that only training 1,000 supervised examples on top of bert @xcite already outperforms bm25, making it very appealing in practical use. based on these results, we perform a deep dive into why a single dense model performs so poorly on these simple questions. we decouple the two distinct aspects of these questions: the entities and the question pattern, and identify what about these 2 nq: natural questions @xcite , tqa: triviaqa @xcite , webq: webquestions @xcite , curatedtrec @xcite . we use the pre-trained models from https://github.com/ facebookresearch/dpr . we end with two investigations of practical solutions towards addressing this crucial problem. first, we consider data augmentation and analyze the trade-off between single-and multi-task finetuning. second, we consider a single fixed passage index and fine-tune specialized question encoders, leading to memory-efficient transfer to new questions. we find that data augmentation, while able to close gaps on a single domain, is unable to consistently improve performance on unseen domains. we also find that building a robust passage encoder is crucial in order to successfully adapt to new domains. we view this study as one important step towards building universal dense retrieval models. dense retrieval on the contrary, dense models @xcite @xcite ) measure similarity using learned representations from supervised qa datasets, leveraging pre-trained language models like bert. in this paper, we use the popular dense passage retriever (dpr) model @xcite as our main evaluation, 3 and we also report the evaluation of realm @xcite in appendix a. dpr models the retrieval problem using two encoders, namely the question and the passage encoders, initialized using bert. dpr uses a contrastive objective during training, with in-batch negatives and hard negatives mined from bm25. during inference, a pre-defined large set of passages (e.g., 21million passages in english wikipedia) are encoded 3 the detailed experimental settings are in appendix b. and pre-indexed-for any test question, the top passages with the highest similarity scores are returned. recently, other advances have been made in improving dense retrieval, including incorporating better hard negatives @xcite , or fine-grained phrase retrieval @xcite . we leave them for future investigation. generalization problem despite the impressive in-domain performance of dense retrievers, their capability of generalizing to unseen questions still remains relatively under-explored. recently, @xcite discover that there is a large overlap between training and testing sets on popular qa benchmarks, concluding that current models tend to memorize training questions and perform significantly worse on non-overlapping questions. amber @xcite test sets are designed to study the entity disambiguation capacities of passage retrievers and entity linkers. they find models perform much worse on rare entities compared to common entities. similar to this work, our results show dense retrieval models generalize poorly, especially on rare entities. we further conduct a series of analyses to dissect the problem and investigate potential approaches for learning robust dense retrieval models. finally, another concurrent work @xcite introduces the beir benchmark for zero-shot evaluation of retrieval models and shows that dense retrieval models underperform bm25 on most of their datasets. ## entityquestions in this section, we build a new benchmark enti-tyquestions, a set of simple, entity-centric questions and compare dense and sparse retrievers. ## towards robust dense retrieval equipped with a clear understanding of the issues, we explore some simple techniques aimed at fixing the generalization problem. ## conclusion in this study, we show that dpr significantly underperforms bm25 on entityquestions, a dataset of simple questions based on facts mined from wikidata. we derive key insights about why dpr performs so poorly on this dataset. we learn that dpr remembers robust representations for common entities, but has trouble differentiating rarer entities without explicitly observing the question pattern during training. we suggest future work in incorporating explicit entity memory into dense retrievers to help differentiate rare entities. numerous recent works @xcite @xcite ) demonstrate retrievers can easily learn dense representations for a large number of wikipedia entities. dpr could also leverage entity-aware embedding models like @xcite or luke @xcite to better recall long-tail entities.