F ollow IR : Evaluating and Teaching Information Retrieval Models to Follow Instructions (2025)

Abstract: Modern Language Models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. While Information Retrieval (IR) models use these LMs as the backbone of their architectures, virtually none of them allow users to provide detailed instructions alongside queries, thus limiting their ability to satisfy complex information needs. In this work, we study the use of instructions in IR systems. First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions. FollowIR repurposes detailed instructions – also known as narratives – developed for professional assessors to evaluate retrieval systems. In particular, we build our benchmark from three collections curated for shared tasks at the Text REtrieval Conference (TREC). These collections contains hundreds to thousands of labeled documents per query, making them suitable for our exploration. Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework. Our results indicate that existing retrieval models fail to correctly use instructions, using them for basic keywords and struggling to understand long-form information. However, we show that it is possible for IR models to learn to follow complex instructions: our new FollowIR-7B model has significant improvements after fine-tuning on our training set.

## introduction modern language models (lms) are extensively tuned to be able to follow user instructions faithfully @xcite @xcite @xcite and safely @xcite . through these capabilities, lms are able to successfully tackle a broad range of tasks @xcite @xcite @xcite , even when not explicitly fine-tuned for them. in contrast to the broader lm community, information retrieval (ir) practitioners and researchers have yet to fully exploit instruction-tuned models. thanks to their ability to effectively estimate semantic similarity between query and documents, lms have been adopted as the main backbone of neural retrieval architectures @xcite @xcite . however, the vast majority of these systems are fine-tuned to operate exclusively as text spans similarity estimators @xcite @xcite @xcite . moving past these ad-hoc search systems to retrieve with instructions would enable support for complex information needs. for example, imagine a researcher seeking to identify papers that must contain numerous qualities to be relevant (from a given venue, using a given class of methods, etc.) while also making sure to avoid conditions that would make it not-relevant (using negative sentiment, using datasets from certain domains, etc.). recent work has started to move towards search with instructions, but this topic is still understudied with only a handful of papers @xcite @xcite . in particular, we find their use of instructions to be narrow: instructions are typically short (fewer than 10 words) and repetitive (only one instruction per dataset e.g., @xcite ; @xcite ; @xcite ; @xcite ). further, these works lack evaluation datasets that explicitly measure instruction following-instead focusing on standard ad-hoc retrieval benchmarks. to address these gaps we introduce followir, which consists of (1) a benchmark that explicitly measures the instruction following ability of retrieval models, and (2) training data that includes diverse and realistic instructions. our key intuition is to leverage instructions developed for professional annotators of ir systems in order to study the capabilities of instruction-following ir models. these instructions are used by annotators to judge document relevance for a given query. fortunately, the ir field is rich with such data, as these instructions-also known as narratives-are created for all queries in any well-constructed ir dataset. in particular, we use narratives developed for shared tasks at the text retrieval conference. these instructions are thorough and complex, including minute details about what makes a document relevant vs not-relevant. thus if annotators can use these trec instructions to annotate document relevance, so should instruction-following retrieval models (example query and instruction pairs are shown in figures 1 and 2 ). we use three deeply-judged trec collections as the basis of our evaluation set: trec robust 2004 @xcite , trec common core 2017 @xcite , and trec news 2021 @xcite . these collections have been thoroughly annotated in order to evaluate recall in retrieval, with hundreds to thousands of documents judged as relevant or not-relevant. we take the instructions given to the professional annotators and alter them slightly, manually re-annotating the relevant documents. we then have paired instructions, which can be used to test how models react to changed instructions: we measure if models update their relevant docs to match the altered instructions. as there are no existing methods to compare pairwise queries in ir, we develop a new evaluation framework to do so, measuring rank-wise score changes (which we call p-mrr) of documents given a pair of different instructions with the same query. results on followir indicate that current models generally fail to follow instructions in retrieval unless they are 3b+ parameters or have not been trained for retrieval. our analysis shows that these failures are due to two phenomena: (1) models are not used to long instructions, and (2) models use the instructions to do keyword search. to further progress in building retrieval models that can understand instructions, we build a training set of real-world human-used instructions and fine-tune a model on them (followir-7b). our results show marked improvement on followir for both standard ir metrics and for p-mrr, indicating a starting point for future progress. in summary, we contribute the following: (1) a benchmark for evaluating instruction following in retrieval (followir) consisting of human annotations on top of three already highly-judged corpora, (2) analysis of why current models fail to understand instructions, and (3) training data for teaching retrieval models to follow instructions along with a new open-sourced ir model, followir-7b, that can handle long instructions in ir. instruction: documents are relevant that show the hubble telescope has produced new data, better quality data than previously available, ...., are relevant if they relied solely on the hubble. documents limited to the problems of the telescope are be irrelevant. details of repairs to the telescope without reference to positive achievements would not ... 2008), or retrieval-augmented generation/reportgeneration @xcite . as part of this process, nist sponsors annotations for these collections. typically, this is done by pooling a set of results (runs) from a variety of retrieval models and then annotating them in rank order until funding runs out. to help facilitate annotation, track organizers provide a narrative (or instruction) for each query that will be given to the annotators-however, ir models are only ever given the query. as evaluating total recall would require annotating every document in the collection for every query (which is not feasible for collections with millions of documents), recall error is tested using post-hoc sampling and annotation. although not every query and document pair can be evaluated, recall for queries is very high. we build off the rigorous evaluation done at trec by starting with several of their collections. instructions for lms instruction-following lms have been popularized by models such as instructgpt @xcite , flan @xcite , and t0 @xcite . they have become a large area of interest for the natural language processing community @xcite @xcite . there has been much work in evaluating if they can generalize to new instructions @xcite , if we can train them to follow instructions without human-annotated data @xcite , and applying them to various domains @xcite @xcite . as the ir community uses lms in their pipelines, we seek to broaden the scope of ir to include instructions, aligning it with the broader nlp community. instructions for retrieval using instructions in retrieval is a nascent area of exploration. @xcite and @xcite were two of the earliest works that trained a retrieval model to use instructions along with the query. however, these instructions are typically very short, such as "retrieve a wikipedia paragraph that answers this question." recent work incorporates instructions in smaller models @xcite @xcite as well as others which use llama @xcite or mistral @xcite as the backbone of a larger retrieval model that can use instructions: gritlm @xcite trains mistral to do both generation and embedding, while @xcite uses mistral for embeddings only. despite this flurry of activity, these efforts do not have an explicit instruction-related retrieval benchmark to evaluate on. instead, they evaluate on standard retrieval benchmark suites such as mteb @xcite and beir @xcite which do not contain instructions. thus, these newer instruction retrieval models hand-write a few instructions, where typically each instruction is applied to an entire dataset, irrespective of the query. this makes these instructions generic: focused only on the task format, format of the "document" (paragraph, sentence, etc.), and the broad domain. note that because of this, no current instructions contain any extra background information or negation @xcite which are commonly found in real-world instructions (see figure 1 for an example of these differences). in work concurrent to ours, @xcite also propose a dataset to evaluate instructions in retrieval models. their dataset uses the ms marco collection @xcite , and differs in sev-eral crucial aspects: it only has one relevant document per query (e.g., sparsely judged), is gpt-4 generated and validated, focuses on the background of the user ("i am a school teaching looking for ..."), and evaluates using the lowest score over n instructions for the same query (measuring robustness). in contrast, we use highly-judged corpora to ensure we can measure recall, use professionally generated instructions, have human-validated relevance judgements, propose a new paired evaluation protocol, and provide a training dataset and model for teaching instruction-following. ## building followir we derive followir from three trec collections: trec news 2021 (derived from the washington post v4 corpus; @xcite , trec robust 2004 (from news articles in disks 4 and 5 collections; @xcite , and trec common core 2017 (from the new york times annotated corpus; @xcite . each of these was professionally assessed to include hundreds of annotations per query (see each of these trec tracks includes instructions for the professional annotators that we now also give to the models. although using these alone can provide some indication of how well models can follow instructions, it doesn't explicitly test their instruction following ability. to more carefully isolate this in our benchmark, we test whether models can respond to small changes in the instruction. to accomplish this, we ask two expert annotators to modify the trec instructions. however, doing this in a naive way would require re-annotating all the document judgements, a non-trivial task requiring immense annotation efforts. 2 instead, we task the annotators with making instructions more specific by including additional constraints that narrow the relevance definition. these transformations cause some previously relevant documents to become non-relevant without introducing any new relevant documents from the pool. therefore, only those documents that were deemed relevant by the original trec assessors need to be re-annotated. this makes the annotation tractable, with only dozens of documents to re-annotate per query instead of a collection of thousands. we annotate a subset of the original trec queries due to cost and overlap: we sample 50 queries from trec robust 2004 that do not overlap with trec common core (as common core used 50 queries from robust04 on a new collection), and 30 queries from trec news 2021. due to differences in retriever quality, if we evaluate by searching over the full collection, each model will retrieve a different number of relevant documents. however, because we evaluate instruction following based on changing the document relevance, models that do poorly in the initial retrieval will have fewer documents which change relevance in the instruction-following evaluation. to rectify this, we instead turn to a reranking task where we include all relevant documents, and use a pool of five models foot_2 to select the top non-relevant documents. to be able to freely distribute the data due to fair-use laws, we chunk the documents into 400-word passages with 200-word overlap and select the highest scoring passages using maxp @xcite . this enables us to distribute our data, which we do by extending the mteb evaluation framework @xcite . ## evaluating instruction following in this section we describe the models we evaluate, their results on followir, and ablations performed to better understand current models. ## teaching instruction following is it possible to improve model performance in following instructions? we show that fine-tuning on a training set of longer instructions can provide a method for doing so. we start by gathering a training set to teach models. we collect all trec narratives (i.e., instructions) from tasks not in fol-lowir, consisting of 1836 pairs of queries and narratives. however, we note that this does not provide any positive or negative documents for fine-tuning. in order to obtain documents for training, we prompt gpt-3.5-turbo-1106 to generate relevant and not-relevant documents, generating roughly two relevant and non-relevant instances per query. the prompts for this experiment can be found in appendix h. however, these synthetic documents are noisy and contains errors w.r.t. the labels-to remedy this, we perform a round of filtering and use the best performing open-source model from when we evaluate this model on followir ( ## conclusion despite the use of lms as the backbone of neural retrieval models, most existing ir models do not take instructions that define document relevance. further, there is no existing resource that measures how well retrieval models can follow instructions. we build a new benchmark that explicitly measures the instruction following ability of retrieval models and find that nearly all retrieval models do not follow instructions, with the exception of larger models (3b+ parameters) or instruction-tuned lms that typically are not used for retrieval. however, we show that it is possible to improve their instruction following ability, and build and release a training corpus for teaching retrieval models to follow instructions. our new model, followir-7b, shows improvement on both standard retrieval metrics as well as in instruction following, which we hope will inspire future work on the topic.