H - L egal KI : A Hierarchical Legal Knowledge Integration Framework for Legal Community Question Answering (2024)

Abstract: Legal question answering (LQA) aims to bridge the gap between the limited availability of legal professionals and the high demand for legal assistance. Traditional LQA approaches typically either select the optimal answers from an answer set or extract answers from law texts. However, they often struggle to provide relevant answers to complex, real-world questions due to the rigidity of predetermined answers. Although recent advancements in legal large language models have shown some potential in enhancing answer relevance, they fail to address the multiple user-specific circumstances, i.e., factual details in questions. To address these issues, we (1) construct the first publicly available legal community question-answering (LegalCQA) dataset; and (2) propose a Hierarchical Legal Knowledge Integration (H-LegalKI) framework. LegalCQA is collected from two widely used legal forums for developing user-centered LQA models. For H-LegalKI, we design a legal knowledge retriever that gathers comprehensive legal knowledge based on both entire questions and individual sentences. And an answer generation model is designed to understand question- and sentence-level factual details and integrate corresponding legal knowledge in a hierarchical way. Additionally, we design a de-redundancy module to remove redundant legal knowledge. Experiments on LegalCQA demonstrate the superiority of our framework over competitive baselines.

## introduction legal question answering (lqa) is expected to address the disparity between the limited number of legal professionals @xcite and the extensive volume of legal issues @xcite . as a significant application of legal artificial intelligence, lqa can provide professional legal advice to assist ordinary individuals in protecting their rights. additionally, for legal professionals, it offers a handy reference to relieve the burdensome work and increase their working efficiency. generally, there are two main research focuses in answering legal questions. the first one involves selecting the correct or optimal answer from a list of potential answers or an answer set. for example, @xcite ; @xcite ; @xcite collect single/multiple-choice questions from judicial examinations, construct qa pairs, and study lqa as a classification problem. @xcite ) construct an answer database, including responses sourced from legal forums, and retrieve the optimal answer for a given question. the other research line focuses on extracting answers from law articles or regulations @xcite , which predict the start and end indices over the relevant legal text. we present examples of qa pairs for both answer selection and extraction in figure 1 . while these works have made progress in lqa, they struggle to effectively and relevantly answer complex legal questions since the answers are actually predetermined. recently, large language models (llms) like gpt-4 @xcite and llamaa @xcite have demonstrated remarkable capabilities in various natural language processing tasks, including qa. several studies train llms with legal corpus and develop a series of specialized legal llms, such as disc-lawllm @xcite , chatlaw @xcite , and wisdominterrogatory 1 . these models can improve the relevance of generated answers to the questions. however, they are primarily trained on professional legal knowledge, which is not aligned with the users' multiple practical circumstances, i.e., factual details described in questions. as a result, they struggle to meet users' specific needs, as shown in our experimental results (4.5 & 4.6). to address these issues, this work concentrates on developing an answer-generation model based on real-world qa data from online legal forums. users seeking legal assistance typically engage with these forums by asking comprehensive questions. and legal professionals contribute highly question-relevant and user-centered answers. by carefully designing a specialized lqa framework to utilize these qa pairs, we aim to bridge the gap between the practical needs of users and professional legal knowledge. specifically, we first collect questions and answers from legal forums and construct a legal community qa dataset, named legalcqa, to address the lack of real-world qa data. legalcqa consists of chinese and english sub-datasets, and covers a wide range of questions with various law categories, e.g., civil law, criminal law, and contract law (more details can be found in 4.1). we show a qa pair in legalcqa at the bottom of figure 1 . this example illustrates that the question includes more factual details (e.g., slander, personal information tort, and civil action) compared to the other two types of questions. the complexity of user-raised 1 https://github.com/zhihaillm questions, often including multiple circumstances and involving various legal knowledge, presents a challenge for traditional open-domain questionanswering methods. effectively capturing these circumstances and integrating relevant legal knowledge to generate high-quality responses is far from trivial. we then propose a novel hierarchical legal knowledge integration (h-legalki) framework for lqa. there are two key components of the proposed framework: (1) a designed legal knowledge retriever is responsible for retrieving hierarchical legal knowledge. specifically, we split the original question q into individual sentences s, and retrieve the involved legal items (e.g., law articles and regulations) for q and all s, respectively. this allows us to acquire more comprehensive legal knowledge to support the answer generation; ( 2 ) a novel answer generation model is designed to make use of the hierarchical information (question-and sentencelevel) from both questions and the retrieved legal items. we fuse the representations of question q to its corresponding legal items to obtain questionspecific legal knowledge representation. and the same process is applied to each sentence. additionally, we design a simple de-redundancy module based on averaging operation to refine legal knowledge representations, as the retrieved legal items may contain duplicate entries. our main contributions are summarized as: ## conclusion in this work, we propose a hierarchical legal knowledge integration (h-legalki) framework to enhance generative lqa. the proposed framework effectively learns multiple factual details in userexpressed questions and integrates comprehensive legal knowledge at both the question and sentence levels. we collect qa pairs from legal forums and construct the first public legal community qa dataset called legalcqa. we conduct plentiful experiments on legalcqa, and the results confirm the effectiveness of h-legalki compared to competitive baselines.