IFIR : A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025)

Abstract: We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.

## introduction the instruction-following ability has become a cornerstone for llms @xcite @xcite @xcite , empowering them to interpret and respond to complex user commands and perform a wide range of user-specific tasks. despite its critical importance, the instructionfollowing capability remains underexplored in the context of information retrieval (ir). current information retrievers struggle to meet the nuanced requirements of users in real-world applications, particularly in specialized fields like law @xcite , healthcare @xcite @xcite , and scientific research @xcite : an illustration of instruction-following ir scenarios explored in this study. the example simulates a legal case search, where the user provides detailed instructions to retrieve relevant legal cases. current ir systems struggle to handle such complex queries. (bottom left): as a result, users have to break down their information needs into simpler, iterative search queries and manually filter the retrieved cases, resulting in a time-consuming and inefficient process. (bottom right): this study focuses on evaluating the progress and limitations of current end-to-end retrieval systems in expert-domain instruction-following ir. 2020; @xcite , where precise and context-aware retrieval is crucial @xcite @xcite . for instance, in legal research, lawyers often search for target cases using detailed instructions that incorporate specific legal criteria, contextual information, and desired outcomes, as illustrated in figure 1 . however, traditional ir systems @xcite @xcite lack the ability to fully understand and process such complex user instructions in an end-to-end manner. consequently, users must break down their complex information-seeking needs into several simpler search queries and manually filter the retrieved cases @xcite , which is timeconsuming and inefficient. on the other hand, existing instruction-following ir benchmarks typically employ simplified instructions, such as single sentence @xcite or a set of keywords @xcite . this simplification in evaluation leads to an incomplete assessment of the model's real-world performance. although concurrent works like fol-lowir @xcite , instruc-tir @xcite and bright @xcite incorporate more complex instructions, they do not establish explicit complexity levels to evaluate retrievers' fine-grained abilities in following instructions. moreover, these studies primarily focus on general domains, leaving the evaluation of instruction-following retrieval in expert domains largely underexplored. in this work, we introduce ifir, a comprehensive benchmark designed to evaluate the instruction-following capabilities of information retrievers, particularly in the context of specialized domains. ifir includes eight subsets covering four specialized domains: finance, scientific literature, law, and healthcare. to provide a more granular evaluation, we create three levels of instruction complexity for each domain, representing a range of real-world information retrieval scenarios in expert domains. ifir includes 2,426 instruction-following queries, each averaging 6.14 ground-truth passages. to ensure the dataset's high quality, we conduct a comprehensive human expert validation during its construction. moreover, recognizing the limitations of traditional evaluation methods in measuring instruction-following ir performance, we introduce a novel llm-based metric, instfol, designed to more accurately assess how well retrievers follow instructions. through extensive experiments on 15 frontier retrievers, including those based on llms, we derive three key findings: (1) bm25 performs relatively well because the instructions in expert domains contain more glossary terms. (2) instruction-tuned retrievers like instructor @xcite do not perform significantly better than their base models, i.e., gtrs @xcite . this demonstrates that current instruction-tuned retrievers may not be suitable for complex instructions. (3) most evaluated models experience performance declines as the complexity of the instructions increases. (4) llm-based retrievers demonstrate more robust performance on both ndcg and instfol, highlighting their potential in managing more complex retrieval tasks in specialized domains. we conclude our main contributions as follows: ## ifir benchmark we introduce ifir, a comprehensive benchmark designed to assess and enhance retrievers' instruction-following capabilities in expert domains. the construction process of ifir, illustrated in figure 2 , employs a semi-automated, human-in-the-loop pipeline that ensures both scala-bility and high quality. specifically, we expand the queries in existing specialized-domain ir benchmark (detailed in §3.1) by incorporating detailed instructions that closely mirror real-world scenarios in relevant expert domains ( §3.2). each example is thoroughly validated by domain experts (details of annotator selection and assigned tasks are provided in ## experiment setup this section outlines the experimental setup of our study. we first introduce the two automated metrics used for evaluating ifir, and then discuss the evaluated retrieval systems. implementation details can be found in appendix b. ## experimental results this section first presents the key experimental results, followed by an in-depth analysis of model performance, an error case study, and a reliability evaluation of the proposed ifir metric. ## reliability analysis of instfol to evaluate the reliability of the proposed llmbased evaluation metric, instfol, we conduct a thorough analysis. our focus is on instfol's core component: the llm's ability to score the relevance between an instruction-following query and a passage. we validate these llm-based relevance scores by comparing them with human evaluation scores. specifically, we randomly sample 400 query-passage pairs and assign each to one domain expert for evaluation. both the human evaluators and the llm (i.e., gpt-4o) are instructed to rate relevance on a scale of 1 to 5. after collecting the scores from both sources, we evaluate their alignment at the instance level. notably, we observe a strong pearson correlation coefficient of 0.704, indicating that instfol can effectively assess a retriever's instruction-following capabilities. ## conclusion we introduce ifir, a novel ir benchmark designed to assess the instruction-following capabilities of retrievers. ifir focuses on domain-specific instructions, reflecting the diverse needs across various fields. our experiments reveal that current instruction-tuned models struggle with long, complex instructions. moreover, as the complexity increases, a noticeable performance decline occurs across all tested retrieval systems. however, llmbased retrievers demonstrate more robust performance and relatively better results compared to other models. this suggests potential solutions for end-to-end specialized-domain retrieval scenarios.