A Two-Stage LLM System for Enhanced Regulatory Information Retrieval and Answer Generation (2025)

Abstract: This technical report describes our methodology for the Regulatory Information Retrieval and Answer Generation (RIRAG) Shared Task, a component of the RegNLP workshop at COLING 2025. The challenge aims to effectively navigate and extract relevant information from regulatory texts to generate precise, coherent answers for compliance and obligation-related queries. To tackle subtask1, we introduce a two-stage approach comprising an initial output stage and a subsequent refinement stage. Initially, we fine-tune the LLaMa-2-7B model using LoRA to produce a preliminary output. This is followed by the application of an expert mechanism to enhance the results. For subtask2, we design specific prompt to facilitate the generation of high-quality answers. Consequently, our approach has achieved state-of-the-art performance on the leaderboard, which serves as a testament to the effectiveness and competitiveness of our proposed methodology.

## introduction regulatory documents, issued by government bodies, detail compliance rules across various areas like environmental standards and data protection. they are complex, comprehensive, and frequently updated, making them challenging to interpret and keep up with. to manage these documents effectively, specialized nlp techniques, such as information retrieval and question answering, are essential for industries facing governance and compliance challenges. at the same time, the rapid development of large language models (llms) @xcite @xcite @xcite has been remarkable, with major breakthroughs in various fields. this progress implies that llms could offer innovative * corresponding author. however, simply deploying llms in missioncritical domains such as healthcare, law, and finance poses unique challenges that go beyond general ai optimization and alignment. a primary concern is the models' propensity to generate plausible but incorrect "hallucinatory" responses, especially in specialized domains where data is limited or complex. furthermore, the vast expansion of online data, along with the substantial resources needed for data annotation and model training, makes it difficult for llms to stay up-to-date. recent innovations are trying to tackle these issues. retrieval-augmented generation (rag) @xcite integrates information retrieval to update static knowledge. chain of thought @xcite prompting has led to task-specific workflows. these are enhanced by parameter-efficient fine-tuning methods like low-rank adaptation (lora) @xcite and reinforcement learning from human feedback (rlhf) @xcite , which improve model performance with minimal parameter increases. however, these methods rely on substantial datasets. in response, an automated question passage generation task for regnlp has been defined @xcite , creating the obliqa dataset with 27,869 questions from the adgm financial regulation documents, providing a rich resource for training and refining llms in regulatory compliance. in this study, we introduce a two-stage framework for regulatory document processing: parameter-efficient fine-tuning of large language models (llms) and a hybrid expert mechanism. our key contributions are as follows: (1) parameter-efficient fine-tuning of llms: we have fine-tuned the general-purpose llama-2-7b model to specialize in domain-specific retrieval tasks. our experiments confirm that this approach significantly enhances the model's ability to accurately retrieve information for regulatory-related queries. (2) hybrid expert mechanism: beyond conventional rank fusion techniques, we propose a novel expert mechanism designed to refine outputs from various experts. this mechanism ensures a higher level of precision and reliability in the final results. ## experiment in this section, we state the details of the experimental implementation. finally, we present the corresponding experimental results. ## conclusion in summary, this paper delves into the effective utilization of llms for regulatory-specific tasks and introduces a hybrid expert mechanism. this innovative approach marries the capabilities of sophisticated llms with the tried-and-true methods of traditional retrieval systems. the result is a significant boost in the efficacy of regulatory information retrieval and answer generation processes. moving forward, our research will focus on uncovering further optimization techniques for these large models and on broadening their application to a more diverse array of regulatory tasks.