Regulatory Question-Answering using Generative AI (2025)

Abstract: Although retrieval augmented generation (RAG) has proven to be an effective approach for creating question-answering systems on a corpus of documents, there is a need to improve the performance of these systems, especially in the regulatory domain where clear and accurate answers are required. This paper outlines the methodology used in our submission to the Regulatory Information Retrieval and Answer Generation (RIRAG) shared task at the Regulatory Natural Language Processing Workshop (RegNLP 2025). The goal is to improve document retrieval (Shared Task 1) and answer generation (Shared Task 2). Our pipeline is constructed as a two-step process for Shared Task 1. In the first step, we utilize a text-embedding-ada-002-based retriever, followed by a RankGPT-based re-ranker. The ranked results of Task 1 are then used to generate responses to user queries in Shared Task 2 through a prompt-based approach using GPT-4o. For Shared Task 1, we achieved a recall rate of 75%, and with the prompts we developed, we were able to generate coherent answers for Shared Task 2.

## introduction regulations are official rules and directives established and maintained by authoritative bodies, such as government or regulatory agencies, to ensure compliance with legal standards. they are crucial for maintaining order, protecting public interests, and fostering fair practices across various industries. due to the extensive range of regulations and the intricate nature of the language used in the regulatory content, comprehending these guidelines can be challenging for both the general public and regulatory professionals. failure to adhere to regulations can result in legal and financial consequences, adversely impacting an organization's reputation and operations. in our submission, we address both shared tasks: passage retrieval (subtask 1) and answer generation (subtask 2). our system design is presented in section 4. evaluation results on the development and test set is shown in section 5. finally, we conclude and discuss the next steps in section 6. ## related work early efforts in regnlp concentrated on pattern matching, rule-based and semantic relation extraction methods. however, devising these patterns and rules can be quite difficult due to the complex nature and style of regulatory texts. traditional information retrieval methods such as best matching-25 (bm25) and term frequency-inverse document frequency (tf-idf) have been widely used for regulatory information retrieval @xcite . but, these methods often struggle with shifts in word distribution and fail to adequately capture semantic similarity between words. regulatory information retrieval using modern machine learning approaches @xcite @xcite ) such as word/document embeddings, recurrent neural networks (rnn), and long short term memory figure 1 : system architecture networks (lstm) are good at modeling language and recognizing semantic similarities among various words and passages. however, they fall short in capturing long-range dependencies. transformerbased approaches have demonstrated notable advancements in retrieval performance, as highlighted by several studies relevant to regulatory retrieval @xcite ) @xcite . however, they still need significant amount of annotated datasets for fine-tuning. with the advent of large language models (llms), we can now accomplish a diverse range of nlp-related tasks without requiring task-specific fine-tuning. ranking is one such area where these models particularly excel. rankgpt @xcite proposes a list-wise ranking methodology @xcite that addresses the issue of the llm's context length by employing a sliding-window technique. passages are ranked within each window, which then shifts incrementally to cover the entire list, ensuring overall re-ranking while staying within the llm's context length limitation. for the task of answer-generation in regulatory domain, lstm and transformer-based architectures have demonstrated significant results as highlighted by @xcite ), but they require domain-specific labeled data for training. recently, llms, through prompt engineering @xcite and rag @xcite have shown promising results in generating coherent and grounded answers in the provided context. in our paper, we use text-embedding-ada-002 from openai for the retrieval followed by rank-gpt for re-ranking, the details of which are elaborated in section 4. the retrieved passages are then used as context within an engineered prompt to generate answers to the user query using gpt-4o. ## dataset the obligation-based question answering (obliqa) dataset @xcite , was specifically developed to support research in regulatory compliance. it includes question-answer pairs derived from passages in regulatory documents provided by the adgm financial authority. these passages were selected individually or identified through topic-based clustering. question-answer pairs were generated utilizing the gpt-4 model. to maintain precision and relevance, the generated questions were meticulously filtered for strong semantic alignment with the corresponding passages. based on the number of passages used to generate the answers, the dataset was categorized into groups, where each group contains different combinations of 1 to 6 input passages per question. the data was subsequently divided into training, development, and test sets, containing 22k, 2.7k, and 2.7k samples, respectively. ## system design the combined architecture of our system for both subtasks is shown in figure 1 and detailed in the following sub-sections. ## results to evaluate the effectiveness of our methodology, we first establish a baseline retrieval using text-embedding-ada-2 for comparison which only returns the initial top-30 results without any reranking. for an initial qualitative analysis, we use a small subset (10%) of the test data. the baseline system achieved 70% recall@10, while re-ranking using rankgpt demonstrated a 5% improvement over this. an example of this improvement is shown in the performance of our method on the full development and test sets for subtask 1 is summarized in on subtask 2, we evaluate the repass metric as described in @xcite . our prompts achieve a high entailment score (e s ), indicating that the answers are well-supported by the source passages. however, performance on obligation coverage (oc s ) is comparatively lower. the overall composite score remains consistent across both datasets and is relatively good, as shown in ## conclusion and future work our retrieval followed by re-ranking methodology demonstrates consistent and relatively good performance on both sets. however, as complexity increases, the effectiveness declines. we intend to investigate contrastive fine-tuning of retrievers to enhance retrieval capabilities and implement corrective-rag for better contextual understanding, thereby delivering more relevant responses. similarly, for answer generation, we observe that there is some degree of contradiction in answers compared to source passages as indicated by relatively high value for c s . we aim to explore mechanisms to detect and resolve these contradictions and improve the obligation coverage (oc s ) by updating the answer generation prompt or potentially using a secondary prompt for refinement.