AUEB -Archimedes at RIRAG -2025: Is Obligation concatenation really all you need? (2025)

Abstract: This paper presents the systems we developed for RIRAG-2025, a shared task that requires answering regulatory questions by retrieving relevant passages. The generated answers are evaluated using RePASs, a reference-free and model-based metric. Our systems use a combination of three retrieval models and a reranker. We show that by exploiting a neural component of RePASs that extracts important sentences (‘obligations’) from the retrieved passages, we achieve a dubiously high score (0.947), even though the answers are directly extracted from the retrieved passages and are not actually generated answers. We then show that by selecting the answer with the best RePASs among a few generated alternatives and then iteratively refining this answer by reducing contradictions and covering more obligations, we can generate readable, coherent answers that achieve a more plausible and relatively high score (0.639).

## introduction the regulatory information retrieval and answer generation (rirag) foot_0 shared task focuses on the development of systems that can effectively retrieve relevant information from regulatory texts to generate accurate answers for obligation-related queries. it is divided into two subtasks: passage retrieval, where systems identify the ten most relevant passages from regulatory documents, and answer generation, which requires synthesizing comprehensive answers from the retrieved passages. we participated with three systems and released our code publicly. 2 each one of them uses a rank fusion @xcite combination of three retrieval models: bm25 @xcite , and two neural domain-specific retrievers, based on a law-and a finance-specific embedding model, respectively. we also apply a neural reranker to the top-n retrieved passages. for answer generation, our first system adversarially exploits the evaluation metric of the task, called repass, by using one of its neural components. specifically, we extract important sentences ('obligations') from the retrieved passages and then concatenate these sentences to get an 'answer'. even though the produced answers may be incoherent and may not answer the question directly, this system achieves a perfect score, much higher than the score of human experts. the second system extends this approach with an llm that generates an answer (for each question) by iteratively reformulating (as parts of an answer) the extracted obligations of the previous system. this results in more readable answers, but performance deteriorates to repass scores below those of the challenge's baseline @xcite . our third system works by a) generating multiple candidate answers and using repass to select the best answer, and b) iteratively refining the selected answer by removing contradictions and adding 'obligation' sentences that increase repass. this system performs worse than the adversarial (first) system, but much better than the baseline, and the answers are coherent and readable. ## passage retrieval all three of our systems use the same passage retrieval, which improves upon the baseline retrieval system of the shared task @xcite in three ways: a) we use domain-specific neural retrieval models, b) we extend the rank fusion approach @xcite to include three models instead of two, and c) we use a reranker. ## answer generation the answer generators of this section use our best retriever (section 3, bm25, vl2, vf2, reranker). ## conclusion we introduced three systems for the rirag shared task. the retrieval backbone of all systems combined bm25 with two domain specific neural retrievers and a reranker. we achieved a near-perfect score with an adversarial system that exploits the neural model for obligation extraction of repass, highlighting the difficulty of developing a robust reference-free metric for rag evaluation. our best non-adversarial system (vrr) first generates multiple alternative answers from the retrieved obligations, selects the alternative answer that maximizes repass, then iteratively improves it by maximizing obligation coverage and minimizing contradictions. this system produces coherent answers, and obtains the highest repass score among competitors that do not exceed human performance (which may be a sign of gaming repass).