1-800- SHARED - TASKS at R eg NLP : Lexical Reranking of Semantic Retrieval ( L e S e R ) for Regulatory Question Answering (2025)

Abstract: This paper presents the system description of our entry for the COLING 2025 RegNLP RIRAG (Regulatory Information Retrieval and Answer Generation) challenge, focusing on leveraging advanced information retrieval and answer generation techniques in regulatory domains. We experimented with a combination of embedding models, including Stella, BGE, CDE, and Mpnet, and leveraged fine-tuning and reranking for retrieving relevant documents in top ranks. We utilized a novel approach, LeSeR, which achieved competitive results with a recall@10 of 0.8201 and map@10 of 0.6655 for retrievals. This work highlights the transformative potential of natural language processing techniques in regulatory applications, offering insights into their capabilities for implementing a retrieval augmented generation system while identifying areas for future improvement in robustness and domain adaptation.

## introduction regulatory documents pose significant challenges for organizations seeking to ensure compliance owing to their complexity and ever-changing nature. it is important for organizations to adhere to regulations to maintain legal compliance. with the recent advances in natural language processing (nlp), there is an opportunity to tackle these issues and automate the process of information retrieval, regulatory comparisons, and compliance verifications. regulatory nlp (regnlp) focuses on improving access to and understanding of regulatory rules and obligations by leveraging nlp techniques. within regnlp, usage of language models for the retrieval of regulatory guidelines for question answering * equal contribution (q/a) has shown great potential @xcite . in light of this, this paper focuses on our submission to the coling 2025 regulatory information retrieval and answer generation (rirag) challenge, involving two key tasks: retrieving top-k relevant passages for the given set of queries and using the relevant passages to formulate answers with language models. our approach enhances the capabilities of semantic retrievals for rirag by fine-tuning an embedding model on positive data pairs and reranking it using lexical retrieval techniques. leser (lexical reranking of semantic retrieval) is a novel hybrid approach that combines dense semantic retrieval with classical lexical reranking for enhanced retrieval performance. it leverages dense embeddings fine-tuned on query-passage pairs and integrates bm25 @xcite scores to improve ranking precision. this dual approach enables robust retrieval in complex regulatory domains, outperforming both pure lexical and semantic models. we test a multitude of open-source models and select the best model for leser. our work contributes to developing specialized retrieval systems for q/a in regulatory domains. ## dataset & task the rirag task aims to enhance the efficiency and accuracy of compliance-related tasks by addressing two critical subtasks in regnlp: passage retrieval (sub-task 1) and answer generation (subtask 2). the first sub-task (1) is to identify and extract the most relevant passages, specifically obligations and related rules, from adgm regulations and guidance documents. building on this, the second sub-task (2) focuses on the ability to geneach instance consists of a natural language question, relevant regulatory passages annotated with documentid and passageid, and contextual group identifiers. obliqa is a multi-retrieval dataset and its distribution is given in ## methodology for the passage retrieval task, our approach leverages a combination of dense and sparse retrieval methods to maximize the relevance and diversity of the retrieved passages as shown in figure 1 . hybrid retrieval approaches combine the strengths of semantic and lexical techniques to enhance retrieval quality. in these methods, semantic embeddings are often used for relevance matching, while lexi-cal models ensure precision by addressing nuances like exact term matching and linguistic variation. such approaches generally merge semantic and lexical scores during retrieval, rather than isolating the two stages. we propose leser (lexical-semantic retrieval), a novel take on hybrid retrieval that uniquely decouples these phases. semantic embeddings retrieve high-recall candidates, which are then reranked lexically for precision. this strict modularity ensures optimal performance tailored to the challenges of regulatory information retrieval. we utilize a dense vector-based search mechanism using the faiss library @xcite , with embeddings generated by fine-tuning an embedding model. a total of 20 top-ranked passages are then retrieved based on cosine similarity scores. to enhance retrieval performance further, we integrate bm25, a classical sparse lexical retrieval method, as a reranking tool. passages retrieved using the fine-tuned embedding model are re-ranked by combining their dense semantic scores with sparse relevance scores generated by bm25 using a weighted aggregation approach, and the top-10 results are passed as context for answer generation. the embedding model was fine-tuned on a dataset derived from obliqa for a maximum of 10 epochs, employing a batch size of 64 and a learning rate of 2 × 10 -5 . the dataset consists of anchor-positive pairs. we used multiple negative symmetric ranking loss (mnsr) for contrastive learning, which treats every in-batch example as a potential negative example for all other queries, maximizing efficiency during training. the "symmetric" aspect means it considers bidirectional relationships (query-to-passage and passage-to-query) to improve the alignment of representations. the dev dataset was used for creating the evaluation dataset for fine-tuning, in order to load the best checkpoint at the end of the training. the model finetuned under this approach includes bge-smallen-v1.5 @xcite , contextual document embeddings (cde) small (morris and rush, 2024), medembed @xcite , mpnet v2 @xcite , and stella 400m english @xcite . the best model is used for retrieving relevant passages using the leser approach. for the answer generation task, we test four opensource models, namely qwen2.5 7b (qwen team, 2024), mistral 7b @xcite , mistral nemo 12b (mistralai, 2024), and gemma-2 9b @xcite . the prompts for answer generation models are designed to incorporate the retrieved passages in sub-task 1 as contexts and the inference is done using batch size of 1. for faster inference, we use unsloth's fastlanguagemodel (unslothai, 2024) for 2x inference performance. for assessing the performance of answer generation, we use repass metric @xcite which measures the overall quality of answer generation using query, retrieved passages, and answer, based on entailment and contradictions scores. for assessing the performance of retrievals, we used recall@10, which measures the proportion of relevant passages retrieved within the top-10 results and mean average precision@10 (map@10), which evaluates the precision of ranked passages. ## results during the fine-tuning phase, various retrieval models were assessed on the test dataset to identify the top-performing systems for the retrieval task ( fine-tuning the dense model improved their performance in recall significantly. bge_mnsr (finetuned with mnsr loss) performed the best with recall@10 of 0.8068, outperforming the baselines model. medembed model, which itself is a finetuned version of bge, performed similarly to bge with recall@10 of 0.8049. however, the semantic retrieval models still lagged behind baseline bm25 in terms of map@10 massively, with medembed having the best map@10 of 0.1108, compared to baseline map@10 of 0.6237. because of its highest recall score, bge_mnsr is implemented in the leser approach. its performance improved massively compared to its previous counterparts. with recall@10 of 0.8201 and map@10 of 0.6655, it outperforms all other models, including the baseline model. this shows the effectiveness of the leser approach in regulatory retrieval systems. for assessing the performance of the answer generation task, we use the answers generated by the model for unseen questions, giving an account of the real-world performance of the system. the performance of models integrating the bge_leser retrieval system with various large language models (llms) was assessed using the repass metric and is shown in ## conclusion our results highlight the significant impact of leveraging hybrid approaches to improve performance in complex retrieval and answer generation tasks. the bge_leser when paired with qwen2.5 7b, demonstrated superior performance in both recall and answer quality, outperforming other models such as mistral 7b, nemo 12b, and gemma-2 9b across multiple metrics. our leser approach demonstrated significant improvements in both recall and precision of retrievals. this progression from traditional retrieval models to advanced llmbased fine-tuning with reranking illustrates the importance of iterative adaptation, allowing models to specialize in retrieving relevant information while simultaneously enhancing their ability to generate coherent, contextually relevant answers. the superior performance of qwen2.5, particularly in the repass evaluation, underscores the potential of integrating fine-tuned retrieval systems with high-performing generative models to address nuanced tasks such as answer synthesis. this work emphasizes the importance of combining robust retrieval mechanisms with effective answergeneration strategies to create ai systems capable of delivering high-quality, actionable insights. integration of sophisticated embeddings and largescale language models within the leser frame-work demonstrates the transformative potential in improving compliance monitoring and regulatory interpretation workflows. moving forward, future research could explore advanced fine-tuning techniques, ensemble models, newer reranking mechanisms, and domain-specific adaptations to further enhance the scalability and interpretability of these systems in regulatory domains.