Referral Augmentation for Zero-Shot Information Retrieval (2024)

Abstract: We propose Referral-Augmented Retrieval (RAR), a simple technique that concatenates document indices with referrals: text from other documents that cite or link to the given document. We find that RAR provides significant performance gains for tasks across paper retrieval, entity retrieval, and open-domain question-answering in both zero-shot and in-domain (e.g., fine-tuned) settings. We examine how RAR provides especially strong improvements on more structured tasks, and can greatly outperform generative text expansion techniques such as DocT5Query and Query2Doc, with a 37% and 21% absolute improvement on ACL paper retrieval, respectively. We also compare three ways to aggregate referrals for RAR. Overall, we believe RAR can help revive and re-contextualize the classic information retrieval idea of using anchor texts to improve the representations of documents in a wide variety of corpuses in the age of neural retrieval.

## introduction zero-shot information retrieval, a task in which both test queries and corpora are inaccessible at training time, closely mimics real-world deployment settings where the distribution of text changes over time and the system needs to continually adapt to new queries and documents. prior work @xcite finds that without access to training on in-domain query-document pairs or taskspecific document relations, most dense models dramatically underperform simple sparse models like bm25, pointing to poor generalization. at the same time, sparse models struggle to reconcile figure 1 : referral-augmented retrieval (rar) improves zero-shot document retrieval across a variety of models and datasets, such as paper retrieval, shown here. while the zero-shot setting lacks querydocument pairs, our key insight is to leverage interdocument relations that provide multiple views of the same information to provide a more comprehensive representations of the concepts in a document. we propose referral-augmented retrieval (rar), a simple technique that augments the text of each document in a retrieval index with passages from other documents that contain citations or hyperlinks to it. this use of inter-document information is reminiscent of google's backrub and pagerank algorithms. in the age of pretrained models, we revisit this classical intuition on new, dense retrievers such as simcse and dpr @xcite , as well as new domains with referral links like the semantic scholar citation graph @xcite and wikipedia entity graph @xcite (more on the recontextualization in appendix section c). title: bert: pre-training of deep bidirectional transformers for language… abstract: we introduce a new language representation model called bert… retrievable with referral-augmented retrieval (ours) retrievable with standard retrieval query: "used pre-trained dense models [citation] for contrastive fine-tuning" referral: recent analysis of masked language modeling pre-training have shown that models such as bert [citation], roberta, and t5 store world knowledge [citation] query: "roberta and t5 are based on recent advances in language modeling [citation] query: "bert [citation]" query: "proposed masked language models [citation] queries referral: we demonstrate that a contrastive objective can be effective when coupled with pre-trained lms like bert [citation], advancing sentence embedding methods ## conclusion we propose a simple method to leverage documentto-document referrals for retrieval and show strong model-and task-agnostic gains over both base retrievers and other text expansion techniques on a variety of tasks. going forward, referrals offer a lightweight way to leverage, edit, and aggregate diverse relational perspectives in retrieval corpora.