Attributed Question Answering for Preconditions in the D utch Law (2024)

Abstract: In this paper, we address the problem of answering questions about preconditions in the law, e.g. “When can the court terminate the guardianship of a natural person?”. When answering legal questions, it is important to attribute the relevant part of the law; we therefore not only generate answers but also references to law articles. We implement a retrieval augmented generation (RAG) pipeline for long-form answers based on the Dutch law, using several state-of-the-art retrievers and generators. For evaluating our pipeline, we create a dataset containing legal QA pairs with attributions. Our experiments show promising results on our extended version for the automatic evaluation metrics from the Automatic LLMs’ Citation Evaluation (ALCE) Framework and the G-EVAL Framework. Our findings indicate that RAG has significant potential in complex, citation-heavy domains like law, as it helps laymen understand legal preconditions and rights by generating high-quality answers with accurate attributions.

## introduction many people encounter civil justice problems at some point in their lives, whether they are disagreements with landlords or issues at work. however, not everyone knows their rights or how to resolve these problems, leaving them unsure of what to do next @xcite . studies have shown that the main obstacles to getting justice are the costs involved and a lack of awareness about legal rights and available options @xcite . this issue is not just local -it is a global problem. over 1.4 billion people around the world have unresolved civil justice needs @xcite , and in a global survey, 43% of respondents said that legal issues had negatively affected their personal lives @xcite . automated legal question answering (qa) could provide affordable assistance to a wide audi-figure 1 : our rag framework for legal attributed qa with an example ence. one concern is that many countries still lack a digital legal aid system, as each country operates under its own legal framework @xcite and in their local language. this requires languagespecific qa solutions, such as those explored by @xcite , which focuses on developing a system capable of answering legal questions in french for belgian law. prior initiatives have been undertaken to assist individuals by creating legal chatbots for various languages, including french, thai, and indonesian @xcite @xcite ). an important requirement of legal qa systems is that they should provide verifiable sources in their responses, so-called attributions, in order to increase the verifiability of the responses. additionally, the answers generated by these systems should be tailored to an individual's specific legal situation and provide detailed information about their legal options. furthermore, the responses should not be too brief, such as simple "yes" or "no" answers @xcite , which fail to capture the complexity of legal issues. in this paper we focus on precondition-related questions. we define a precondition to refer to the specific requirements, criteria, or circumstances that must be fulfilled before a specific action, event, decision, or outcome can legally occur or be finalized. some examples of precondition-related questions are: "when is a student eligible for student financing?" and "what are the requirements for entering into a marriage?". the answer to the first question contains preconditions such as having a dutch nationality. the latter question can be answered with the precondition of being at least 18 years old. we address attributed qa (aqa) for the dutch law. our aim is to answer legal questions with a tailored answer, including attributions to the relevant law article. attributions have the form of references to specific documents, in our case articles of the dutch law. to this end, we create and publish a dutch legal qa dataset, consisting of 102 question-answer pairs with attributions to dutch law articles. we implement and evaluate a retrieval augmented generation (rag) pipeline that generates informative long-form answers to dutch law questions, where each generated answer contains a list of attributions (references) to sources. our approach is illustrated in figure 1 . our contributions are as follows: the answers have been verified by a legal expert on legal correctness. we publicly release our code and dataset at https://gitlab.com/normativesystems/ flintfillers/aqa_preconditions . ## experiments we first conduct experiments to select the best retriever component for our rag pipeline. once the best-performing retriever for each scenario is identified, we proceed to conduct experiments on the entire rag pipeline using the best-performing retriever based on its recall@k score. ## discussion the results show that our rag system can generate fluent and correct answers with an 83.0% hit-rate. the answers are often highly coherent with the ground truth, and the models are capable of citing their sources accurately. looking at the retrievers, there are substantial differences. dragon consistently underperformed our baseline model, bm25. we hypothesize that this might be because dragon is a dense retriever trained solely in the english language, lacking multilingual capabilities. the e5 models, which were the best retriever models in our rag system, were also trained using contrastive learning. following e5, the hybrid model splade, and the dutch-trained sbert performed best. this is an interesting finding, especially considering that splade was trained only for the english language, leading us to hypothesize that its performance could be attributed to its partly sparse characteristics, enabling lexical overlap. while the models (4, 5, and 6 in the results on the generation show that the proprietary models scored higher on all evaluation metrics than the open-source models. there are several explanations for the substantial difference, but the main one probably lies in the parameter sizes between the models. although the number of parameters for the proprietary models used in our work remains undisclosed, it is reasonable to assume that they are significantly larger than the open-source models that we have used for our work which are relatively small models ranging between the performances are evaluated on fluency, correctness and citation through nine evaluation metrics: g-eval coherence (coh), g-eval fluency (flu), rouge-l (rou), meteor (met), g-eval consistency (con), g-eval relevance (rel), precision (p), recall (r), and hitrate@5 (hit) ## conclusion in this paper, we create and evaluate a retrieval augmented generation (rag) pipeline for attributed question answering for the dutch law, generating long-form answers to precondition questions. we experiment with several state-of-the-art retrievers and generators. for evaluating our pipeline, we create and release a dataset containing 102 legal qa pairs with attributions, as well as an automated evaluation framework suited to this task. the results show that our rag system can generate fluent and largely correct answers with an 83.0% hit-rate. future work includes an extension of the dataset with other document types such as jurisdictions, include more retrievers such as a multilingual hybrid retriever and compare the results from the evaluation framework with human judgements.