Grounded Answers from Multi-Passage Regulations: Learning-to-Rank for Regulatory RAG (2025)

Abstract: Regulatory compliance questions often require aggregating evidence from multiple, interrelated sections of long, complex documents. To support question-answering (QA) in this setting, we introduce ObliQA-MP , a dataset for multi-passage regulatory QA, extending the earlier ObliQA benchmark (CITATION), and improve evidence quality with an LLM–based validation step that filters out ~20% of passages missed by prior natural language inference (NLI) based filtering. Our benchmarks show a notable performance drop from single- to multi-passage retrieval, underscoring the challenges of semantic overlap and structural complexity in regulatory texts. To address this, we propose a feature-based learning-to-rank (LTR) framework that integrates lexical, semantic, and graph-derived information, achieving consistent gains over dense and hybrid baselines. We further add a lightweight score-based filter to trim noisy tails and an obligation-centric prompting technique. On ObliQA-MP, LTR improves retrieval (Recall@10/MAP@10/nDCG@10) over dense, hybrid, and fusion baselines. Our generation approach, based on domain-specific filtering plus prompting, achieves strong scores using the RePAS metric (CITATION) on ObliQA-MP, producing faithful, citation-grounded answers. Together, ObliQA-MP and our validation and RAG systems offer a stronger benchmark and a practical recipe for grounded, citation-controlled QA in regulatory domains.

## introduction regulatory documents are long, complex, and highly structured, often requiring professionals to synthesize information from multiple sections across documents to answer questions. unlike typical information retrieval tasks, relevant evidence is frequently distributed across semantically related but non-contiguous passages in multiple documents. this poses challenges for retrieval augmented generative question answering systems: they must identify not only directly answer-bearing text but also supporting context scattered across long complex documents. to support progress in this domain, the regulatory information retrieval and answer generation (rirag) task @xcite was introduced, along with the obliqa dataset @xcite , which provides synthetic qa pairs grounded in real regulatory obligations. however, the initial formulation of obliqa primarily focused on single-passage retrieval and relied on limited validation procedures, reducing its applicability to more realistic multi-passage scenarios. we address these limitations by introducing obliqa-mp, a multi-passage extension of obliqa requiring evidence from multiple documents, and by refining the validation pipeline. we incorporate a large language model (llm)-based validation step that significantly improves evidence quality by filtering out approximately 20% of passages incorrectly retained by earlier nli-based checks. building on this dataset, we conduct a systematic study of retrieval methods tailored for regulatory texts. specifically, we explore a feature-based learningto-rank (ltr) framework that integrates lexical, semantic, and graph-based signals, and evaluate its effectiveness against strong baselines. our main contributions are: retrieval: sparse, dense, fusion, and ltr classical sparse retrieval (e.g., bm25) remains competitive on legal/regulatory corpora, yet fusion and learning-to-rank (ltr) often yield stronger topk quality. reciprocal rank fusion (rrf) provides a simple, effective ensemble over heterogeneous runs @xcite . in-domain dense retrievers (e.g., e5 @xcite ) complement lexical signals, and hybrids (bm25+dense) frequently outperform either alone. feature-based ltr with lambdamart @xcite and lightgbm @xcite exploits lexical overlap, run scores/ranks, and corpus signals; we adopt a two-stage setup that gathers bm25/dense/rrf candidates and re-ranks with ltr tailored to regulation. legal rag systems and benchmarks legal rag is emerging, with benchmarks and systems emphasizing evidence-grounded answers (e.g., legalbench-rag @xcite ; interpretable statute qa @xcite ; casebased and adaptive pipelines @xcite ; kg-augmented rag @xcite ). these works are steps towards practical legal assistants but largely target case law or statutes rather than regulatory obligations. our focus is complementary: ranking and selecting regulatory passages for grounded answers in a multi-passage multidocument setting. we construct obliqa-mp, a multi-passage extension of the obliqa dataset @xcite , starting from the original generation pipeline-which includes structured regulatory documents, topic-based rule clustering, and llmbased question generation. we merge newly generated multi-passage questions and their associated passages with the original obliqa dataset, and retain only those questions that are linked to multiple passages, resulting in 13,191 candidate examples. to ensure the quality and relevance of the supporting evidence, we validate each questionpassage pair using the gpt-4.1-2025-04-14 model. the model receives the following prompt: you are validating if a passage answers a question. reply with: the final obliqa-mp dataset consists of 2,976 multi-passage qa pairs. we randomly split this dataset into training, validation, and test sets using a 70/15/15 ratio. listing 1: example qa pair from obliqa-mp showing multi-passage evidence across documents. the llm-based validation step introduced in obliqa-mp is intended to increase the precision of supervision by addressing a systematic failure mode of nli-only validation. in the original obliqa pipeline, passages were treated as premises and questions as hypotheses under a lightweight nli model (nli-deberta-v3-xsmall) @xcite . however, nli-only screening can admit false positives: a passage may linguistically entail the question yet fail to provide extractable, citation-backed answer spans (e.g., generic or templated compliance statements)-a problem amplified in multipassage settings where evidence is distributed. in obliqa-mp we therefore prompt gpt-4 with citation control to label each candidate passage as directly connected (contains answer spans), indirectly connected (supportive context), or not connected. this finer-grained validation reveals that 20.46% of the 31,037 evaluated passages are not connected despite having previously passed nli filtering. moreover, only 2,976 out of 13,191 candidate multi-passage questions contain at least one directly connected passage and at least one additional supporting passage, underscoring the necessity of stricter validation for dispersed-evidence queries. as a result, the retained qa pairs are grounded in semantically related passages that also provide explicit, citation-backed evidence sufficient to support answer generation; we nonetheless treat the llm check as a precision-oriented filter rather than expert adjudication and acknowledge that a targeted expert audit remains future work. ## feature-based learning-to-rank for regulatory rag our method builds on the standard rag pipeline-retrieve, re-rank, and generate (fig. 1 ). the key novelty lies in a feature-based learning-torank retriever, combined with a lightweight scorebased filter to prune noisy candidates before answer generation. ## conclusion we introduced obliqa-mp, a multi-passage regulatory qa dataset, and a stricter llm-based validation that removes incomplete or off-target evidence more effectively than prior nli filters. building on this resource, we presented a retrieve-rerank-generate pipeline that couples a feature-based ltr retriever with lightweight score-based filtering and an obligation-centric prompt with mandatory [p#] citation control. our experiments show that multi-passage retrieval is markedly harder than single-passage obliqa, yet ltr consistently outperforms strong sparse, dense, and fusion baselines on retrieval metrics. more importantly, under a common, deterministic generation setup, ltr plus filtering and prompt discipline yields the best answer quality on repass-raising entailment and lowering contradictions relative to prior bm25 + gpt baselines on obliqa-mp. future work will focus on exploring hybrid retrievers that combine symbolic and neural representations, developing more robust generation pipelines, and designing evaluation metrics that align more closely with expert-level legal reasoning and compliance practices.