DIRAS : Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025)

Abstract: Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information when answering queries that need an integrated analysis of information (e.g., Tell me good news in the stock market today.)? To address these concerns, RAG developers need to annotate information retrieval (IR) data for their domain of interest, which is challenging because (1) domain-specific queries usually need nuanced definitions of relevance beyond shallow semantic relevance; and (2) human or GPT-4 annotation is costly and cannot cover all (query, document) pairs (i.e., annotation selection bias), thus harming the effectiveness in evaluating IR recall. To address these challenges, we propose DIRAS (**D**omain-specific **I**nformation **R**etrieval **A**nnotation with **S**calability), a manual-annotation-free schema that fine-tunes open-sourced LLMs to consider nuanced relevance definition and annotate (partial) relevance labels with calibrated relevance scores. Extensive evaluation shows that DIRAS enables smaller (8B) LLMs to achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs, and is helpful for real-world RAG development.

## introduction rag has become one of the most popular paradigms for nlp applications @xcite . one core phase of rag systems is information retrieval (ir), which leverages cheap retrievers to filter relevant information and thus save llm inference costs. however, ir can be a performance bottleneck for rag @xcite . both leaving out important relevant information (low recall) as well as including excessively related but irrelevant information (low precision) may query domain-specific definition diras-fine-tuned small llm: relevance annotator user input output documents for all documents relevant irrelevant well-calibrated relevance ranking using relevance score binary verdict: e.g. yes ## diras pipeline the diras pipeline is illustrated in fig. 2 . it comprises three steps: sampling a subset of (query, document) pairs for training data creation, obtaining relevance definitions for queries, and distilling relevance annotation ability from teacher to student llms. ## optimizing relevance annotation to train a highly performant student llm m s for relevance annotation, we proceed in three steps. first, we annotate a novel task-specific dataset for our evaluation, chatreportretrieve ( § 3.1). second, we assess design choices and related work baselines to find the best-performing strategy for our relevance-annotation data creation with the teacher model m t ( § 3.2). third, we optimize the fine-tuning of student llms m s by investigating implementation variants ( § 3.3). ## real-world applications having established and benchmarked the design choices of diras student llms, we now investigate their real-world application capabilities. first, we showcase how the diras m s can assist ir annotation in a real-world setting, leveraging the climretrieve dataset @xcite ( § 4.1). second, we show the diras pipeline is also applicable for general domain qa ( § 4.2). ## conclusion in this work, we introduce the diras pipeline to fine-tune open-source llms to calibrated annotators. the diras approach has two significant advantages: (1) it is case-specialised allowing the incorporation of domain-specific knowledge into definitions, and (2) it helps to efficiently label a huge amount of documents with calibrated relevance scores.