RAG ulator: Effective RAG for Regulatory Question Answering (2025)

Abstract: Regulatory Natural Language Processing (RegNLP) is a multidisciplinary domain focused on facilitating access to and comprehension of regulatory regulations and requirements. This paper outlines our strategy for creating a system to address the Regulatory Information Retrieval and Answer Generation (RIRAG) challenge, which was conducted during the RegNLP 2025 Workshop. The objective of this competition is to design a system capable of efficiently extracting pertinent passages from regulatory texts (ObliQA) and subsequently generating accurate, cohesive responses to inquiries related to compliance and obligations. Our proposed method employs a lightweight BM25 pre-filtering in retrieving relevant passages. This technique efficiently shortlisting candidates for subsequent processing with Transformer-based embeddings, thereby optimizing the use of resources.

## introduction the complexity, volume, and ever-changing nature of regulatory documents present unique challenges in governance, compliance, and legal frameworks across various sectors. addressing these challenges demands specialized approaches in natural language processing (nlp) to enable effective management and utilization of regulatory content. the retrieval and answer generation (rirag) shared task as part of the regnlp workshop focuses on building systems that can effectively navigate and extract relevant information from regulatory texts to generate precise, coherent answers for compliance and obligation-related queries. the task is divided into two main subtasks: (1) passage retrieval -given a regulatory question, participants must develop systems to identify and retrieve the most relevant passages, specifically obligations and related rules, from adgm regulations and guidance documents; this paper is structured as follows. section 2 discusses existing work on regnlp. section 3 describes the obliqa dataset, and section 4 introduces the evaluation metrics. section 5 describes our approach to develop rirag system, which first retrieves relevant passages for a given query and secondly generates an answer from these passages, and presents our evaluation of both steps. section 6 reports the results of the applied approaches. our primary contributions in this work can be summarized as follows: ## related work the integration of retrieval-augmented generation (rag) techniques and associated technologies hold potential for enhancing @xcite . by capitalizing on advancements in nlp and information retrieval systems, these methods can alleviate the difficulties posed by intricate and ever-evolving regulatory documents, thereby streamlining access to such documents and boosting compliance efficiency. rag has significantly improved the accuracy, efficiency, and trustworthiness of llms by integrating external, contextually relevant and up-to-date information @xcite . notable approaches include: self-rag @xcite improves response quality by incorporating self-reflection mechanisms. @xcite introduced an efficient qa system that combines local knowledge base search with generative context-based qa. @xcite proposed an algorithm for subgraphs extraction from a knowledge graph based on question entities and answer candidates. the proposed technique boosts hits@1 scores of the pre-trained text-to-text language models by 4-6%. @xcite demonstrated how a system for argument retrieval can significantly improve the quality of a language model-based question answering system for comparative questions. all aforementioned methods highly improve the trustfulness of the qa system and minimize hallucinations @xcite . lms often struggle to pay enough attention to the input context and generate texts that are unfaithful or contain hallucinations. to mitigate this issue, context-aware decoding (cad) @xcite was introduced, which follows a contrastive output distribution that amplifies the difference between the output probabilities when a model is used with and without context. however, these studies do not consider regulatory documents so we are interested in testing the ability of rag methods for solving the qa task for regulatory questions. ## dataset the obligation-based question answering dataset (obliqa), specifically compiled for competition organizers, is based on regulatory documents provided by abu dhabi global markets (adgm). adgm serves as the authority overseeing financial services within the uae's free economic zones. obliqa has been developed as a multi-document, multi-passage question answering dataset, designed specifically to advance the field of regulatory natural language processing (reg-nlp). it comprises 27,869 questions along with their associated source passages. each question may have from 1 to 6 relevant passages. the dataset is categorized into groups with varying distributions of relevant passages for the questions. following this categorization, the entire dataset is split into three sections: training (comprising 22,295 questions), testing (featuring 2,786 questions), and development (consisting of 2,888 questions). ## evaluation to evaluate the retrieval stage in rirag, we primarily use recall@10 as the metric. this is because we depend on the retrieval module to capture as much relevant information as possible, while the task of filtering out noise is left to the answer generation module. the answer generation subtask is evaluated by a reference-free regulatory passage answer stability score (repass). repass designed to assess generated answers within regulatory compliance contexts. this metric evaluates answers through the lens of three pivotal criteria: (1) each sentence within an answer must find support in a corresponding sentence from the source passage(s); (2) answers are required to exclude any sentences that introduce contradictions to the information established in the source passage(s); (3) comprehensive coverage is essential; answers must encapsulate all obligations delineated in the source passages, ensuring that every critical regulatory obligation is accurately mirrored in the response. ## regulatory information retrieval and answer generation task the pipeline of the proposed approach can be found in appendix d. ## results for evaluation we used the labeled test split of obliqa and not the hidden evaluation split that was introduced in @xcite . the labeled test split contains 2,786 question-passage pairs, while the hidden evaluation has only 446 pairs. ## conclusion in this paper, we have described the system we submitted for the rirag challenge at the reg-nlp workshop, specifically concentrating on developing a qa system tailored to the regulatory domain. we proposed a simple yet effective qa pipeline. our study highlighted that lightweight bm25 pre-filtering can efficiently retrieve candidate passages for more resourceful fusion using transformer-based embeddings. we demonstrated that techniques proven successful in general domains may not directly translate to the regulatory domain, as seen with the unsuccessful application of contextualization.