MST - R : Multi-Stage Tuning for Retrieval Systems and Metric Evaluation (2025)

Abstract: Regulatory documents are rich in nuanced terminology and specialized semantics. FRAG systems: Frozen retrieval-augmented generators utilizing pre-trained (or, frozen) components face consequent challenges with both retriever and answering performance. We present a system that adapts the retriever performance to the target domain using a multi-stage tuning (MST) strategy. Our retrieval approach, called MST-R (a) first fine-tunes encoders used in vector stores using hard negative mining, (b) then uses a hybrid retriever, combining sparse and dense retrievers using reciprocal rank fusion, and then (c) adapts the cross-attention encoder by fine-tuning only the top-k retrieved results. We benchmark the system performance on the dataset released for the RIRAG challenge (as part of the RegNLP workshop at COLING 2025). We achieve significant performance gains obtaining a top rank on the RegNLP challenge leaderboard. We also show that a trivial answering approach *games* the RePASs metric outscoring all baselines and a pre-trained Llama model. Analyzing this anomaly, we present important takeaways for future research. We also release our [code base](https://github.com/Indic-aiDias/MST-R)

## introduction automated q&a systems hold tremendous potential in not only improving access to, and comprehension of regulatory obligations, but also help organizations achieve regulatory compliance with reduced costs and latency. currently, compliance workflows are largely manual and organizations need to employ a large number of costly subject matter experts. high recall is especially critical in this domain, as the cost of false negatives i.e. missing crucial regulatory information can lead to severe financial penalties, legal repercussions, and reputational harm. retrieval-augmented generation (rag) offers a promising solution but their performance falls short when frags (frozen rags or rags with pre-trained (frozen) components) are directly applied since regulatory documents utilize specialized, domain-specific terminology and nuanced legal semantics. domain-specific adaptations are needed to make these systems viable. this paper primarily focuses on the retriever part of the system, presenting a simple domain adaptation approach to significantly improve the performance of the retriever part of the system by fine-tuning on the target domain. our contributions include: (a) mst-r a multi-stage retrieval system domain adapted using a multi-stage fine-tuning approach. (b) state of the art retrieval performance on the rirag challenge with a improvement of 12.1% in recall@10 and 23% in m ap @10 compared to the bge baseline from @xcite analysis of the repass metric with a solution that games, with important takeaways. ## prior work passage retrieval is a critical step in rag systems. early methods relied on sparse representations such as tf-idf and bm25 @xcite . dense encoders @xcite using late interaction @xcite , enable document embedding caching. in contrast, approaches such as @xcite @xcite use querydocument interaction for nuanced semantic alignment but incur higher computational costs due to per-sample processing. recently, hybrid search algorithms @xcite combining lexical patterns with semantic relationships between queries and passages have emerged. advanced retrieval systems like re2g @xcite employ multi-level architectures to optimize performance and efficiency. inspired by the above, we propose a multi-level architecture which fuses the results of a variety of approaches from the literature to leverage their complementary strengths. the retrieval stage in qa systems is typically evaluated using metrics like recall@k and mean average precision (map@k). for answer quality, metrics such as bleu and rouge focus on ngram statistics but neglect semantic equivalence in abstractive generation. semantic-focused metrics @xcite @xcite emphasize alignment and coherence but lack comprehensive coverage, especially in regulation domains. to address this, repass @xcite enforces coverage of all obligations within relevant passages, ensuring both relevance and regulatory compliance. ## evaluation we now present the results of the proposed system on the obliqa dataset @xcite . a simple pre-processing step was applied to remove section headers, etc. by filtering out passages with fewer than 10 tokens. following the guidelines in the challenge, we used recall@10 and m ap @10 to evaluate retrieval performance and the repass metric to evaluate the goodness of the generated answer. answer generation performance is presented in astonishingly, pc achieves a rather high score of 0.947, 130% relative improvement in the repass metric over the llama model. even when we convert the entire answer into a single line, we get a rather high performance of 0.801. these results highlight the limitations of the repass metric for in evaluating answer quality. ## analysis: the repass metric rirag uses standard metrics to evaluate the retrieval performance but defines a novel metric called repass to evaluate the answer. repass is reference-free, using neither the ground truth answer nor the input question. in this section, we analyze the properties of the metric itself. ## conclusion the regulatory domain presents significant challenges due to its complex language and contextual requirements needing good domain adaptation strategies. we presented a domain-adapted, multilayered retrieval system showing significant performance gains. while we didn't adapt the performance of the answer generator (leaving it for future work), we present the need for better evaluation metrics as a precursor to engineering better answering models. limitations: considerable research is needed to engineer per-formant answering systems for regulatory domains. while we present an engineered system, it has not been comprehensively evaluated to be deployment-ready. on the other hand, our paper is indicative of the research needed for creating such systems.