NUST Nova at RIRAG 2025: A Hybrid Framework for Regulatory Information Retrieval and Question Answering (2025)

Abstract: NUST Nova participates in RIRAG Shared Task, addressing two critical challenges: Task 1 involves retrieving relevant subsections from regulatory documents based on user queries, while Task 2 focuses on generating concise, contextually accurate answers using the retrieved information. We propose a Hybrid Retrieval Framework that combines graph-based retrieval, vector-based methods, and keyword matching BM25 to enhance relevance and precision in regulatory QA. Using score-based fusion and iterative refinement, the framework retrieves the top 10 relevant passages, which are then used by an LLM to generate accurate, context-aware answers. After empirical evaluation, we also conduct an error analysis to identify our framework’s limitations.

## introduction the regulatory information retrieval and answer generation (rirag) shared task focuses on advancing question answering (qa) in regulatory compliance. participants develop systems to retrieve relevant information and generate precise answers to complex compliance queries, addressing the critical need for interpreting specialized regulatory language in domains like legal research and policy analysis. traditional information retrieval (ir) methods like bm25 @xcite excel at keywordbased document retrieval but struggle with the nuanced, context-dependent language of regulatory texts @xcite . vector-based retrieval, leveraging document embeddings, shows promising results but faces challenges with domain-specific terminology and maintaining relevance @xcite . similarly, graph-based retrieval excels in regulatory contexts but suffers from scalability issues and handling ambiguous or incomplete * corresponding author: mehwish.fatima@seecs.edu.pk data @xcite @xcite . these limitations underscore the need for hybrid approaches to enhance precision and scalability in regulatory text retrieval @xcite . to address the challenges of regulatory qa, we propose a hybrid retrieval framework with multimethod scoring to enhance passage retrieval precision. the framework combines three models: (1) neo4j, which structures queries and passages into a graph for initial relevance extraction, (2) bm25 for keyword matching, and (3) faiss for ranking passages based on semantic similarity. these models are fused through score-based fusion, refining results by combining bm25 and faiss outputs with those from neo4j. this hybrid approach ensures accurate retrieval of the top 10 passages. finally, the llama model generates context-aware, regulatorycompliant answers, effectively handling domainspecific terminology and complex relationships. ## conclusion this work presents a hybrid framework combining vector-based, graph-based, and keyword-matching techniques to enhance regulatory information retrieval and answer generation. the approach significantly improves relevance and contextual accuracy, especially in handling domain-specific content. preliminary results show improvements over baseline methods, with promising retrieval performance. however, answer generation results require refinement, highlighting the need for further enhancement. future work includes exploring different llms or fine-tuning them for regulatory data and incorporating summarization techniques to optimize answer generation and extending graph-based retrieval to operate on entire documents rather than individual passages.