Bilingual BSARD : Extending Statutory Article Retrieval to D utch (2025)

Abstract: Statutory article retrieval plays a crucial role in making legal information more accessible to both laypeople and legal professionals. Multilingual countries like Belgium present unique challenges for retrieval models due to the need for handling legal issues in multiple languages. Building on the Belgian Statutory Article Retrieval Dataset (BSARD) in French, we introduce the bilingual version of this dataset, bBSARD. The dataset contains parallel Belgian statutory articles in both French and Dutch, along with legal questions from BSARD and their Dutch translation. Using bBSARD, we conduct extensive benchmarking of retrieval models available for Dutch and French. Our benchmarking setup includes lexical models, zero-shot dense models, and fine-tuned small foundation models. Our experiments show that BM25 remains a competitive baseline compared to many zero-shot dense models in both languages. We also observe that while proprietary models outperform open alternatives in the zero-shot setting, they can be matched or surpassed by fine-tuning small language-specific models. Our dataset and evaluation code are publicly available.

## introduction open access to legal information is considered a fundamental right according to the charter of fundamental rights in the european union @xcite . effective retrieval models are an essential component to ensuring this right, as they allow laypeople and legal professionals to efficiently search through vast amounts of legal information. in countries like belgium, where laws are available in multiple languages (e.g. french and dutch), the need for high-performance legal retrieval models becomes even more crucial, as they require equal accessibility to relevant legal material regardless of the language in use. ## related work in the last few years, the field of legal nlp has gained increased interest, leading to the development of a growing number of datasets for research. in this section, we focus specifically on datasets that address the task of legal retrieval grounded in legal provisions, including documents, statutory articles, and cases. cail2018 @xcite ) is a dataset designed for legal judgment prediction in chinese, released as part of the chinese ai and law challenge foot_3 . it contains over 2.68 million chinese criminal cases, linked to 183 law articles and 202 charges. one of the subtasks from this challenge involved predicting relevant law articles based on the factual descriptions of specific cases. following this, the cail2019-scm dataset @xcite focuses on similar case matching with 8,964 case triplets (in which two cases are similar) sourced from the supreme people's court of china. zhong et al. (2020) released jec-qa, a question answering dataset based on the chinese bar exam. the dataset contains 26,365 multiple-choice questions, along with 3,382 chinese legal provisions. the aila competitions @xcite introduced datasets for precedent and statute retrieval from indian law, with content in english. for each year, around 50 queries were linked to relevant documents in retrieval corpora containing 197 statutes and around 3,000 prior cases. similarly, coliee @xcite @xcite competitions include the task of statute article retrieval from provided datasets. for each year, the datasets contain around 100 test questions from the japanese legal bar exams, labeled with relevant articles from the japanese civil code, translated into english. the provided training sets include up to 1000 questionarticle pairs. chen et al. (2023) introduced equals, a dataset containing 6,914 question-article-answer triplets, with a corresponding retrieval corpus of 3,081 chinese law articles. the question-answer pairs were collected from a free chinese legal advice forum, then revised and further annotated by senior law students. similarly, stard @xcite introduced 1,543 queries from the general public, with a retrieval corpus of 55,348 chinese statutory articles. gerlayqa @xcite consists of around 21,000 legal questions from laymen paired with answers from legal professionals and grounded in paragraphs from german law books. most related to our work is bsard (louis and spanakis, 2022); a statutory article retrieval dataset which contains over 1,100 legal questions from belgian citizens and around 22,600 belgian law articles as the retrieval corpus. lleqa @xcite complements bsard with answers from legal experts, along with an additional 760 legal questions and 5,308 statutory articles. while lleqa is a more extensive resource than bsard, the latter is available under less restrictive terms foot_4 and does not require a separate user agreement 6 . the resources presented above support the training, evaluation, and benchmarking of retrieval models across different legal domains and languages, highlighting the need for tailored approaches in each jurisdiction. contributing to the growing field of legal nlp, we introduce bbsard, a bilingual dataset built on bsard which offers parallel belgian law articles in both french and dutch, along with legal questions translated from french to dutch. in addition to providing a reliable benchmark for the retrieval task in dutch, bbsard aims to address challenges of legal retrieval in multilingual jurisdictions. ## dataset as mentioned, we base our work on the bsard dataset @xcite , extending it to the dutch language by adding the corresponding articles and questions. we discuss the procedure in the following sections. ## experimental setup this section describes our experimental setup used to benchmark the retrieval performance of a selection of models on bbsard. we mostly reuse the codebase from bsard @xcite , making modifications where necessary to accommodate the retrieval models to the specific requirements of our experiments. below we describe the models, data processing steps, and evaluation metrics used in our experiments. ## results and discussion in this section, we present the performance of various retrieval models evaluated on the dutch and french subsets of the bbsard dataset (see tables ## conclusions and future work in this paper we presented bbsard, the bilingual version of the bsard dataset @xcite . to curate bbsard, we scraped parallel dutch and french articles from the online justel database and translated the bsard questions into dutch. in addition to a parallel bilingual legal corpus, bbsard offers a much-needed retrieval benchmark for the dutch language, allowing for more accurate and reliable evaluation of dutch retrieval models. based on our dataset, we conducted extensive benchmarking of the retrieval task (ranking passages by their relevance to a given query) for dutch and french, both in zero-shot and fine-tuned scenarios. these experiments confirm the status of simple lexical methods like bm25 as strong baselines, the superiority of closed-source commercial models like voyage and openai in zero-shot setting, and the possibility of outperforming them via fine-tuning small language-specific models like robbert and flaubert. we also observed an overall advantage for french compared to dutch, in both zero-shot and fine-tuning scenarios. we hope that our work encourages and facilitates the development of better dutch retrieval models in the legal domain, which are an essential part of popular llm-based methods like rag. in the future, we would like to first improve bb-sard's quality by manually checking/correcting all translated questions, and then expand our work beyond the legal domain by curating and providing a more comprehensive benchmark for the retrieval task in dutch. another interesting research avenue considers the cross-lingual training potentials offered by a bilingual parallel dataset. in our experiments, we observed that xlm-roberta performs better in dutch when finetuned for 50+50 epochs on french+dutch data, compared to 100 10 282 gb and 270 gb before filtering/cleaning. epochs on the dutch subset. this suggests the possibility of leveraging the bilingual structure for additional gains in performance, specially for the lower-resource language, but to examine and explore its real significance more experiments need to be conducted.