NUST Omega at RIRAG 2025: Investigating Context-aware Retrieval and Answer Generations-Lessons and Challenges (2025)

Abstract: NUST Omega participates in Regulatory Information Retrieval and Answer Generation (RIRAG) Shared Task. Regulatory documents poses unique challenges in retrieving and generating precise and relevant answers due to their inherent complexities. We explore the task by proposing a progressive retrieval pipeline and investigate its performance with multiple variants. Some variants include different embeddings to explore their effects on the retrieval score. Some variants examine the inclusion of keyword-driven query matching technique. After exploring such variations, we include topic modeling in our pipeline to investigate its impact on the performance. We also study the performance of various prompt techniques with our proposed pipeline. With empirical experiments, we find some strengths and limitations in the proposed pipeline. These findings will help the research community by offering valuable insights to make advancements in tackling this complex task.

## introduction regulatory documents, issued by governmental bodies, define the rules and standards for legal compliance across industries. these texts are often lengthy and complex, requiring specialized expertise to interpret, with non-compliance carrying heavy penalties @xcite . advancements in nlp have led to the emergence of regulatory natural language processing (regnlp), a multidisciplinary subfield aimed at simplifying access to and interpretation of regulatory texts @xcite . retrieval-augmented generation (rag) @xcite leverages llms by integrating external knowledge sources, enabling up-to-date, domain-adaptable capabilities @xcite @xcite . the rirag shared task consists of two phases: (1) retrieval and (2) answer generation. accurate retrieval, crucial for effective generation, employs techniques like chunking, query expansion, metadata annotation @xcite , and topic modeling to segment regulatory texts for improved precision @xcite . advanced prompting strategies, such as few-shot prompting @xcite and chain-of-thought (cot) @xcite , further enhance response quality in the generation phase. in our approach to the rirag task, we explore multiple methodologies. we begin with metadatabased keyword retrieval and refine it using topic modeling for coherent segmentation. for answer generation, we leverage few-shot and cot prompting to enhance accuracy and coherence. our results emphasize the critical role of retrieval quality in boosting generation performance while highlighting limitations that pave the way for future research. ## progressive retrieval pipeline we propose a pipeline, progressive retrieval pipeline (proreg), for this shared task by adopting an iterative and structured approach. figure 1 illustrates the architecture of proreg foot_0 . ## conclusion in this study, we have explored the applicability of rag for regulatory documents. we approach the task by systematically exploring the performance of embedding models, keyword supported query matching, and topic modeling in compliance with contextual keywords. key lessons from our experiments include the significance of embedding models with respect to the retrieval. the unsuccessful outcome of query matching led us to approach the problem by ingesting topic modeling in the pipeline. moving forward, focusing on sub-topic modeling could provide deeper insights. additionally, fine-tuning the model may improve performance, but experimenting with a more hierarchical rag pipeline could unlock significant potential.