Shared Task RIRAG -2025: Regulatory Information Retrieval and Answer Generation (2025)

Abstract: This paper provides an overview of the Shared Task RIRAG-2025, which focused on advancing the field of Regulatory Information Retrieval and Answer Generation (RIRAG). The task was designed to evaluate methods for answering regulatory questions using the ObliQA dataset. This paper summarizes the shared task, participants’ methods, and the results achieved by various teams.

## introduction regulatory compliance is a critical but highly complex domain, requiring organizations to interpret and adhere to a wide range of rules, standards, and obligations. these tasks are traditionally laborintensive and involve meticulous analysis of all regulatory documents to ensure compliance. the growing volume and complexity of regulations has made manual processes increasingly unsustainable. addressing these challenges necessitates innovative solutions to automate regulatory compliance tasks. the regulatory information retrieval and answer generation (rirag) focuses on automating two core processes: retrieving relevant regulatory information and generating concise, accurate answers to compliance-related questions. by combining information retrieval and answer generation, rirag provides a framework to streamline compliance workflows and enhance organizational efficiency. to foster collaboration and innovation in this emerging field, we organized the rirag-2025 shared task. this shared task aims to benchmark and advance methodologies for regulatory information retrieval and answer generation, bringing together academic and industrial researchers to address real-world compliance challenges. ## overview of teams' methodologies the participating teams in the rirag-2025 shared task employed diverse methodologies to address the challenges posed by the two subtasks. this section provides an overview of the approaches used by the teams, categorized by subtask. ## teams' evaluation results the evaluation of team submissions was conducted separately for both subtasks. the evaluation was based on a hidden subset of the obliqa dataset consisting of 446 unseen questions. subtask 1 (information retrieval): the highest r@10 and m@10 scores were achieved by indic aidias with their first submission, scoring 0.787 and 0.663, respectively. teams nlp-mjr (r@10: 0.731, m@10: 0.602) and team: 1-800 (r@10: 0.705, m@10: 0.562) also performed strongly in the retrieval subtask. ## lessons from rirag-2025 the rirag-2025 shared task attracted a substantial number of participating teams from both academia and industry. this strong participation underscores the rapid growth and increasing interest in the regnlp field. a significant observation during the task was the limited integration of the hierarchical regulatory documents provided into the participants' approaches. although the teams primarily used the obliqa dataset, the rich interconnected structure of the entire set of regulatory documents was underutilized. regulatory rules often refer to or build on one another, and understanding these relationships is crucial for generating accurate and comprehensive answers. future shared tasks can address this perhaps by providing annotated examples of rule connections and offering detailed guidelines to help participants incorporate these relationships into their system designs. in the answer generation subtask, we employed repass, a metric specifically designed for rirag. however, we observed two critical areas for improvement. firstly, repass is currently limited in its ability to evaluate verbatim reproduction of retrieved passages, which can affect the depth and originality of generated answers. secondly, it lacks a mechanism to evaluate the fluency and cohesion of generated answers. to address these shortcomings, future iterations could enhance repass by incorporating penalties for excessive verbatim text and integrating components that assess linguistic quality. specifically, we will explore the inclusion of semantic similarity thresholds to ensure that generated answers synthesize information rather than directly copying it. additionally, we intend to incorporate llm-based evaluations to measure fluency and cohesion, providing qualitative assessments of the generated text. ## conclusion the rirag-2025 shared task showcased innovative approaches to tackling the challenges of regulatory information retrieval and answer generation. by leveraging the obliqa dataset and a robust evaluation framework, participants were able to explore diverse methodologies, from hybrid retrieval systems combining sparse and dense models to ad-vanced generative techniques supported by prompt engineering and post-processing. while the task brought to light many promising methodologies, it also revealed areas for future exploration. the shared task has set a benchmark for further research in this domain, fostering collaboration between academic and industrial researchers and driving advancements in the automation of regulatory compliance tasks.