Task-aware Retrieval with Instructions (2023)

Abstract: We study the problem of retrieval with instructions, where users provide explicit descriptions of their intent along with their queries to guide a retrieval system. Our solution is a general-purpose task-aware retrieval system, trained using multi-task instruction tuning and can follow human-written instructions to find relevant documents to a given query. We introduce the first large-scale collection of 37 retrieval datasets with instructions, BERRI, and present TART, a single multi-task retrieval system trained on BERRI with instructions that can adapt to a new task without any parameter updates. TART advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger. We further introduce a new evaluation setup, X2-Retrieval, to better reflect real-world scenarios in which diverse domains and tasks are pooled. TART significantly outperforms competitive baselines in this setup, further highlighting the effectiveness of guiding retrieval with instructions.

## introduction information retrieval (ir) is the task of finding relevant documents from a large collection of texts to fulfill a user's information need, typically expressed in the form of a textual query @xcite . the notion of relevance from the user's perspective (i.e., intent) can be amorphous @xcite , and a query alone may not fully capture user information needs @xcite . as illustrated in figure 1 (top), given the same query, "implementing batch normalization," users' intents can be diverse (e.g., find code snippets or paragraph-length answers). most existing work tries to learn those implicit intents from labeled data (e.g., pairs of queries and relevant documents), yielding separate models for despite active research in language models (lms), instruction-following has not been systematically explored in retrieval, partly due to the lack of annotated resources. to facilitate research in retrieval with instructions, we introduce berri (bank of explicit retrieval instructions), a collection of approximately 40 retrieval datasets with diverse instructions in a unified format, covering 10 diverse domains. each task has on average 3.5 diverse instructions annotated by experts, following our novel instruction schema for retrieval tasks. we showcase the benefit of berri to train tart (task-aware retriever), a multi-task retrieval system that learns to follow instructions to perform diverse tasks. we employ two widely explored architectures: tart-dual, a dense dual-encoder architecture that retrieves documents based on the similarity of independently encoded query succeeded by instructions and document embeddings; tart-full, a cross-encoder architecture that calculates probabilities of a document being relevant to the query according to the instruction. we train tart leveraging hard negative samples and new instruction-unfollowing negative samples. the tart models, particularly tart-full yields state-of-the-art results on two popular zero-shot retrieval benchmarks, beir @xcite and @xcite , outperforming systems using three times more parameters @xcite @xcite as well as task-specific retrievers trained on millions of automatically generated examples @xcite . we further introduce a new evaluation setup, x 2 -retrieval (cross-task cross-domain retrieval), where a system needs to handle queries with diverse intents and find relevant documents from a largescale, cross-domain pooled corpus, simulating challenges in real-world retrieval applications. tart outperforms other state-of-the-art methods, demonstrating its effectiveness in this under-explored setting, leveraging explicit textual intents. in summary, our contributions are as follows: zero-shot training of retrievers. recent neural retrievers @xcite @xcite show their superiority over term-based retrievers (e.g., bm25; robertson and zaragoza 2009) across domains when training data is abundant @xcite @xcite . due to the high annotation cost, improving neural retrievers in zero-shot settings is an active area of study. pre-training neural retrievers @xcite and training a single retriever on large-scale supervised datasets such as ms @xcite show improvements in transferring to related retrieval tasks @xcite @xcite , while they often struggle with tasks unlike those used for training @xcite . to address this, several work @xcite trains customized retrievers for each task using unlabeled corpora, leveraging another model to automatically generate training data @xcite . it often requires running massive lms and training separate retrievers, resulting in slow and costly adaptation. concurrent to our work, @xcite trains a single dual-encoder model trained on embedding tasks including retrieval tasks with instructions. ## tart: multi-task instructed retriever we now present tart (task-aware retriever) trained on berri via multi-task instruction-tuning, leveraging our unified task-aware schema. ## experiments we evaluate tart on zero-shot retrieval (section 5.1) and our new more challenging evaluation setup, x 2 -retrieval (section 5.2). ## conclusion this paper lays the foundation for building a general-purpose task-aware retriever that can follow natural language instructions. we introduced a new setup, retrieval with instructions, to model users' intents explicitly. we presented berri, the first large-scale retrieval dataset with expert-written annotations. building upon berri, we trained the first instruction-following retrieval system by massive multi-task instruction-tuning, tart advances the state of the art on two zero-shot retrieval benchmarks beir and lotte as well as on our newly introduced challenging evaluation setup.