Simple and Efficient ways to Improve REALM (2021)

Abstract: Dense retrieval has been shown to be effective for Open Domain Question Answering, surpassing sparse retrieval methods like BM25. One such model, REALM, (Guu et al., 2020) is an end-to-end dense retrieval system that uses MLM based pretraining for improved downstream QA performance. However, the current REALM setup uses limited resources and is not comparable in scale to more recent systems, contributing to its lower performance. Additionally, it relies on noisy supervision for retrieval during fine-tuning. We propose REALM++, where we improve upon the training and inference setups and introduce better supervision signal for improving performance, without any architectural changes. REALM++ achieves ~5.5% absolute accuracy gains over the baseline while being faster to train. It also matches the performance of large models which have 3x more parameters demonstrating the efficiency of our setup.

## introduction open-domain question answering (odqa) @xcite is a task that aims to answer questions directly using a large set of documents without being given a specific document. these systems generally employ a "retriever-reader" based approach where a document retriever first retrieves a subset of evidence documents and a document reader processes the documents to identify the correct answer @xcite . recently, dense retrieval methods @xcite @xcite have improved over sparse retrievers like bm25 @xcite and made training these systems end-to-end by leveraging approximate mips search @xcite . realm is an end-to-end model, pre-trained on masked language modeling, that can be finetuned for qa tasks without relying on external sources like bm25 for supervision like dpr @xcite . hence, it is simple and easier to train but is not competitive to pipeline alternatives like dpr. when finetuning, it uses a single gpu making it not directly comparable in scale to dpr which uses more resources for better optimization. due to limited resources it is inefficient, taking more than a day to train. additionally, it uses distant supervision for the retriever in the form of passages containing the target answer leading to ambiguous supervision for training. in this paper, we present a study of realm aimed at understanding and improving its limitations. we find that realm is significantly underoptimized and improve the training by scaling the system through (i) using exact mips search, (ii) introducing larger batch training, and (iii) scaling the reader to process more documents. we further address the noisy distant retrieval supervision by augmenting the training sets with human-annotated evidence passages. since such human annotations are not available for every dataset and is expensive to obtain, we show that models trained with strong supervision transfer well to other datasets where such annotations are not available, indicating the benefits beyond a single annotated dataset. incorporating our best findings, we show that an improved version of realm, which we call realm++ achieves ∼5.5% absolute accuracy improvements over the baseline on multiple odqa benchmarks while processing 4x more examples/sec and outperforms all prior methods of similar parameter regime. further, it shows comparable performance to models with 3x more parameters. our results demonstrate that scale and supervision play an important role in odqa systems highlighting the need for careful comparisons across systems in odqa and for taking scale and efficiency into account in addition to performance when reporting results. experiments test em dev em dev r@10 realm (guu et al., 2020) 40.4 38.2 -realm (ours) 39.4 35.6 68.8 +scale 42.8 37.9 69.5 +scale+ps (10 docs inf) 43.2 38.6 69.9 +scale+ps (100 docs inf) 44.8 38.6 69.9 +scale+rerank 42.3 37.4 67.5 when finetuning 1 , the retriever is trained using distant supervision with passages containing the target answer as positive and the reader is supervised using human annotated short answer spans @xcite . we follow the same design and optimization setup of finetuning realm on qa. we explore the limits of various experiment choices by introducing simple changes to the training and inference setup. ## realm ++ based on the findings in §2, we incorporate the best working components: (i) scaling at training §2.1 (ii) better passage supervision §2.2 and (iii) scaling reader during inference §2.4 to establish an improved realm model, which we call realm++. we study the effect of realm++ on three datasets nq, web questions (wq) @xcite , and curated trec (ct) @xcite . as wq and ct do not have evidence passage annotations, we use them to study the transfer capabilities of the passage supervised nq model. realm++ outperforms models of similar size and is comparable to large models: ## conclusion in this work, we present a study of a denseretrieval qa system, realm, and identify key limitations in its experimental setup. we find that realm is significantly undertrained and we improve realm by introducing simple changes to its training, supervision, and inference setup. we propose realm++ which incorporates our best findings and show that it can achieve significant improvement over prior methods and perform comparably with models with 3x more parameters.