Answering legal questions from laymen in G erman civil law system (2024)

Abstract: What is preventing us from building a NLP system that could help real people in real situations, for instance when they need legal advice but don’t understand law? This question is trickier than one might think, because legal systems vary from country to country, so do the law books, availability of data, and incomprehensibility of legalese. In this paper we focus Germany (which employs the civil-law system where, roughly speaking, interpretation of law codes dominates over precedence) and lay a foundational work to address the laymen’s legal question answering empirically. We create GerLayQA, a new dataset comprising of 21k laymen’s legal questions paired with answers from lawyers and grounded to concrete law book paragraphs. We experiment with a variety of retrieval and answer generation models and provide an in-depth analysis of limitations, which helps us to provide first empirical answers to the question above.

## introduction as the legal system defines one of the fundamental pillar of democracy, it should be easily accessible for any member of society, regardless of their social background or education. a recent survey of comprehensibility of legal texts in germany foot_0 revealed that although searching the internet was the primary choice for 74% of the respondents, most consulted a lawyer afterward claiming that online resources were not helpful enough. moreover, 31% stated they avoided consulting law books because they do not understand them. unfortunately, unlike in case-law systems, in germany's civil-law system law books and their interpretation are the only source of "truth" in legal matters. as a result, individuals must trade off the urgency of their legal problem with the costs of consulting a lawyer, which favors those with more financial resources. leveraging nlp tools to address legal question answering has been an active research topic. however, existing works have been focusing on questions asked by experts, such as lawyers or legal scholars @xcite @xcite . to the best of our knowledge, very few considered the perspective of a layman, that is an ordinary person without any legal expertise and, most importantly, without skills in understanding legalese, the legal jargon @xcite . to fill this research gap, we asked the following research questions. first, how can we best setup empirical research in the domain of legal qa by laymen in a civil-law system? in particular, how can we create a large dataset that (a) contains laymen's questions that are (b) answered by expert lawyers and also (c) grounded in existing law books? second, to which extent the current transformer-based retrieval models and text-generation models are able to tackle the task? third, and most importantly, what are the fundamental challenges of this task preventing success of the current state-of-the-art approaches? this paper presents a ground work for addressing these research questions. we present ger-layqa, a new dataset consisting of 21,538 actual examples for legal german layperson questions accompanied by valid lawyer answers grounded to law books. we then benchmark a variety of pre-trained and/or fine-tuned large language models and semantic retrieval systems. we also conduct in-depth quantitative and qualitative analyses of the results to show the current limitations and where further research is necessary. all datasets, source codes and models are publicly available at https://github.com/ trusthlt/eacl24-german-legal-questions . ## related work there has been a growing interest in using nlp to answer legal questions across diverse languages and legal domains. to this end, @xcite conducted a study on the vietnamese legal system, using a semantic similarity retrieval mechanism to retrieve relevant legal paragraphs in response to questions. @xcite concluded that both extractive and abstractive question answering are still largely unexplored in legal texts. @xcite provided a short survey of services providing legal aid (in english) by navigating users to fill out a predefined form or using a chat-like interface. in the german legal domain, @xcite built one of the first qa datasets by asking lawyers to manually formulate questions for various case law documents. they then further compared the performance of sparse and dense methods for information retrieval and found that the pre-trained bert model they used could not outperform the sparse retrieval methods. @xcite extended their research based on their previously created dataset and focused on developing a system to answer questions by retrieving sub-sections of relevant case law documents for a given query. our paper differs from these related works significantly. firstly, we refer to legal paragraphs (actual sections in law books) instead of legal case documents (e.g., court judgments). secondly, the existing systems's replies only refer to some significant subsections of documents in their dataset, while we aim to provide an easily understandable answer. lastly, while @xcite relied on a dataset they manually crafted together with legal experts, we use real-world examples instead. ## introducing the gerlayqa dataset to the best of our knowledge, no dataset that contains real-life examples of legal questions in everyday language, with answers provided by legal experts and, ideally, references to relevant law paragraphs exist, at least not in german as general availability of various legal-related datasets is sparse to non-existent. therefore, we created such a dataset ourselves by utilizing qa pairs from a german legal online forum frage-einen-anwalt.de where laypersons pose their queries and, for a small fee, receive answers from legal experts. ## experiments our approach to answering laymen's legal questions mimics the way humans would solve a legal issue. it consists of a two step approach as shown in figure 4 : document retrieval, which aims to find the most relevant laws to a given question text, and answer generation, which should generate an easily understandable answer to the layperson's question. ## analysis and discussion this section presents a detailed manual look at the results of our two stages. with this, we aim to identify possible challenges the models faced, influencing their performances. ## conclusion this paper introduced gerlayqa, the first german dataset designed for laypeople seeking legal advice. our dataset comprises real-life examples of questions asked by laypeople, lawyers' corresponding answers, and relevant legal paragraphs. by including all relevant aspects when working on legal cases, we have created a comprehensive database for our and future research that aims to assist laypeople in seeking legal guidance. we experimented with a two-step qa pipeline, similar to the workflow used by lawyers. we found that all tested models delivered only moderate performances. for the most hindering aspects of the models' performances, we identified their difficulties in understanding german legal texts. since the models were not trained on legalese, creating proper semantic embeddings for this formal language is challenging. as a result, especially for document retrieval, using such embeddings to compare the semantic meanings for paragraphs in legalese and questions in everyday language only produces moderate retrieval results. furthermore, the models struggle in essential tasks to grasp legal nuances and understand legal correlations due to insufficient training when providing a legal answer. future work training a bespoke model for german laymen's and expert legal text analysis, similar to the legal-bert @xcite , might improve the accuracy and efficiency of laymen legal qa. further, including more legal expertise in evaluation can be highly beneficial. it would allow us to assess the model's outputs with a legal background rather than just comparing them to the provided gold standard using various automatic metrics. for document retrieval, this legal knowledge would provide us with a more accurate means of determining the relevance of each retrieved paragraph to the query instead of relying on binary labels. similarly, for answer generation, legal experts could aid in validating the legal soundness and binding nature of the answers rather than solely relying on statistical metrics like rouge and bert-score.