H y PA - RAG : A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications (2024)

Abstract: While Large Language Models (LLMs) excel in text generation and question-answering, their effectiveness in AI legal and policy applications is limited by outdated knowledge, hallucinations, and inadequate reasoning in complex contexts. Retrieval-Augmented Generation (RAG) systems improve response accuracy by integrating external knowledge but struggle with retrieval errors, poor context integration, and high costs, particularly in interpreting AI legal texts. This paper introduces a Hybrid Parameter-Adaptive RAG (HyPA-RAG) system tailored for AI legal and policy, exemplified by NYC Local Law 144 (LL144). HyPA-RAG uses a query complexity classifier for adaptive parameter tuning, a hybrid retrieval strategy combining dense, sparse, and knowledge graph methods, and an evaluation framework with specific question types and metrics. By dynamically adjusting parameters, HyPA-RAG significantly improves retrieval accuracy and response fidelity. Testing on LL144 shows enhanced correctness, faithfulness, and contextual precision, addressing the need for adaptable NLP systems in complex, high-stakes AI legal and policy applications.

## introduction the development of large language models (llms) capable of processing and generating human-like text has made significant strides in recent years, such as openai's gpt models @xcite , google's gemini models @xcite and open alternatives such as the llama series @xcite . these models, which store vast amounts of information within their parameters through extensive pre-training, have demonstrated impressive performance in various tasks, including text generation and question-answering across multiple do-mains @xcite @xcite . despite this, llms encounter limitations when applied to specialised fields such as law and policy. these include the rapid obsolescence of their knowledge, which is confined to the data available up to the last pre-training date @xcite , and hallucinations, where the model produces text that seems plausible but is factually incorrect or misleading, driven by internal logic rather than actual context @xcite . empirical studies show that many artificial intelligence (ai) tools designed for legal applications overstate their ability to prevent hallucinations @xcite . indeed, instances of lawyers being penalised for using hallucinated outputs in court documents (fortune, 2023; business insider, 2023) underscore the need for reliable ai question-answering systems in law and policy naturally, retrieval-augmented generation (rag), which enhances llms by incorporating external knowledge, is proposed as a solution. however, this comes with its own challenges. common failure points @xcite include missing content, where relevant documents are not retrieved, leading to unanswered questions; context limitations, where retrieved documents are not effectively incorporated into the response generation process due to limitations in consolidation strategies; and extraction failures, where models fail to extract accurate information from the provided context due to noise or conflicting data. furthermore, advanced retrieval and generation techniques, such as query rewriters and llm-based quality checkers, often result in increased token usage and costs. to address these challenges, this research integrates three key components (see figure 6 in appendix a.2 for a flow overview and figure 1 for the system design): (1) adaptive parameter selection using a domainspecific query complexity classifier to minimise unnecessary token usage, (2) a hybrid search system combining dense, sparse, and knowledge graph retrieval methods to enhance retrieval accuracy. these elements are combined to create a hybrid parameter-adaptive rag system tailored specifically to mitigate the common rag failure points, for the ai policy domain, using nyc local law 144 as the primary corpus. we also provide a streamlit demo for testing purposes. ## background and related work recent llm advancements have impacted fields like law and policy, where language complexity and large text volumes are prevalent @xcite @xcite . llms have been used for legal judgment prediction, document drafting, and contract analysis, showing their potential to improve efficiency and accuracy @xcite @xcite . techniques like fine-tuning, retrieval augmentation, prompt engineering, and agentic methods have adapted these models for specific legal tasks, enhancing performance in summarisation, drafting, and interpretation @xcite . retrieval-augmented generation (rag), as formalized by lewis et al., enhances pre-trained seq2seq models by integrating external knowledge through indexing, retrieval, and generation stages, improving response specificity and accuracy @xcite . rag systems complement llms by combining sparse (e.g., bm25) and dense (e.g., vector) retrieval techniques, using neural embeddings to refine document retrieval and produce grounded, high-quality responses @xcite @xcite . to address the limitations of naive rag, such as insufficient context and retrieval inaccuracies, advanced techniques have been developed, including hybrid retrieval methods, query rewriters, and rerankers to refine relevance @xcite @xcite . hybrid retrieval combines bm25 with semantic embeddings to balance keyword matching and contextual understanding, improving outcomes @xcite @xcite . additionally, knowledge graph retrieval and composed retrievers enhance accuracy and comprehensiveness in document retrieval @xcite @xcite . recently, rag systems have advanced from ba-sic retrieval to dynamic methods involving multisource integration and domain adaptation @xcite . innovations like self-rag and kg-rag improve response quality and minimize hallucinations through adaptive retrieval and knowledge graphs @xcite . various frameworks have been developed to evaluate rag systems, including ragas, which uses reference-free metrics like faithfulness and relevancy @xcite . giskard @xcite assesses performance using synthetic qa datasets, while ares utilizes prediction-powered inference (ppi) with specialized llm judges for accurate evaluation @xcite . ## system design the hybrid parameter-adaptive rag system, depicted in figure 1 , integrates vector-based text chunks and a knowledge graph of entities and their relationships to enhance retrieval accuracy. the system employs a hybrid retrieval process, combining sparse (bm25) and dense (vector) methods to retrieve an initial top-k set of results. these results are refined using reciprocal rank fusion based on predefined parameter mappings. simultaneously, a knowledge graph retriever identifies relevant triplets, with retrieval depth and keyword selection dynamically adjusted according to query complexity. results from both bm25 and vector methods are fused again to produce a final optimised set of k chunks. optional components include a query rewriter, which generates reformulated queries to improve retrieval. the rewritten queries fetch additional chunks, which are de-duplicated and fused to maintain uniqueness. an optional reranker can further refine chunk ranking if needed. the final set of selected chunks and knowledge graph triplets are then processed within the llm's context window for more accurate, contextually relevant responses. this framework is implemented in two variations: without knowledge graph retrieval, known as parameter-adaptive (pa) rag, and with knowledge graph retrieval, termed hybrid parameter-adaptive (hypa) rag. ## ai legal and policy corpus local law 144 (ll144) of 2021, enacted by the new york city department of consumer and worker protection (dcwp), regulates automated employment decision tools (aedts). this paper uses a 15-page version of ll144 that combines the original law text with enforcement rules published by the dcwp. as an early ai-specific law, ll144 is included in the training data of foundational models like gpt-4 and gpt-4o, whose understanding of the law is confirmed manually through targeted prompting and serves as baselines in this research. ll144 presents significant challenges for ai compliance due to its unique combination of qualitative and quantitative requirements. unlike most ai legal and policy texts, which are predominantly qualitative, ll144 integrates detailed definitions and procedural guidelines with quantitative compliance metrics. this structure complicates interpretation and retrieval, often exceeding the capabilities of traditional llms and rag systems. furthermore, ai laws and policies are frequently revised while moving through the legislative process, making them impractical for pre-training and finetuning and therefore requiring a robust method for integrating changes. ## performance evaluation the evaluation process starts by generating custom questions tailored to ai policy and legal questionanswering, then introduces and verifies evaluation metrics (see evaluation section of figure 6 in appendix a.2). for reproducibility, the llm temperature is set to zero for consistent responses and all other parameters are set to defaults. ## chunking method we evaluate three chunking techniques: sentencelevel, semantic, and pattern-based chunking. sentence-level chunking splits text at sentence boundaries, adhering to token limits and overlap constraints. semantic chunking uses cosine similarity to set a dissimilarity threshold for splitting and includes a buffer size to define the minimum number of sentences before a split. pattern-based chunking employs a custom delimiter based on text structure; for ll144, this is "\n §". figure 3 shows that pattern-based chunking achieves the highest context recall (0.9046), faithfulness (0.8430), answer similarity (0.8621), and correctness (0.7918) scores. sentence-level chunking, however, yields the highest context precision and f1 scores. semantic chunking performs reasonably well with increased buffer size but generally underperforms compared to the simpler methods. further hyperparameter tuning may improve its effectiveness. these findings suggest that a corpusspecific delimiter can enhance performance over standard chunking methods. for subsequent experiments, we adopt sentencelevel chunking with a default chunk size of 512 tokens and an overlap of 200 tokens. ## query complexity classifier to enable adaptive parameter selection, we developed a domain-specific query complexity classifier that categorises user queries, each corresponding to specific hyper-parameter mappings. our analysis of top-k selection indicated different optimal top-k values for various question types, as shown in figure 7 (appendix a.4). ## overall results and discussion our analysis shows that adaptive methods generally outperform fixed baselines, particularly in improving faithfulness and answer quality. incorporating adaptive parameters such as query rewrites and reranking enhances the system's ability to provide accurate and relevant responses. while adding a reranker improves correctness, it can slightly reduce the overall correctness score, suggesting a trade-off between precision and answer quality. the introduction of a knowledge graph maintains correctness but can add complexity, potentially lowering overall response quality. however, combining adaptive parameters with a reranker proves effective in maximizing the proportion of correct responses, even if it doesn't lead to the highest scores in all metrics. overall, these findings highlight the importance of adaptivity and careful parameter tuning to balance different performance aspects, enhancing the system's capability to handle varied and complex queries effectively. ## limitations and future work this study has several limitations that suggest areas for future improvement. correctness evaluation is limited by reliance on a single evaluator familiar with the policy corpus. averaging a larger quantity of human evaluations would improve reliability. additionally, our knowledge graph construction process may be improved. for instance, using llm-based methods for de-duplication and/or custom cypher query generation to improve context retrieval and precision. furthermore, our parameter mappings were not rigourously validated quantitatively. further evaluation of parameter selections could provide better mappings as well as upper and lower bounds to performance. the classifier was trained using domain-specific synthetically generated data -which, though we inject significant noise, may harbour the llm's own unconcious biases in terms of structure -possibly limiting the generalisability of the classifier on unseen user queries. also, more classification categories e.g. 4 or 5-class, would permit more granular parameter selections and potentially greater efficiency improvements. another limitation is that while ll144 is included in the gpt models' training data, subsequent minor revisions may affect the accuracy of these baseline methods. integrating human feedback into the evaluation loop (see figure 4 ) could better align metrics with user preferences and validate performance metrics in real-world settings. future work should also consider fine-tuning the llm using techniques like rlhf @xcite , rlaif @xcite , or other preference optimisation methods @xcite . further, refining the query rewriter @xcite and exploring iterative answer refinement (asai et al., 2023) could enhance metrics like relevancy and correctness. ## ethical and societal implications the deployment of the hybrid parameter-adaptive rag (hypa-rag) system in ai legal and policy contexts raises critical ethical and societal concerns, particularly regarding the accuracy, reliability, and potential misinterpretation of ai-generated responses. the high-stakes nature of legal information means inaccuracies could have significant consequences, highlighting the necessity for careful evaluation. we emphasize transparency and reproducibility, providing detailed documentation of data generation, retrieval methods, and evaluation metrics to facilitate replication and scrutiny. the environmental impact of nlp models is also a concern. our system employs adaptive retrieval strategies to optimize computational efficiency, reduce energy consumption, and minimize carbon footprint, promoting sustainable ai development. our findings enhance the understanding of rag systems in legal contexts but are intended for research purposes only. hypa-rag outputs should not be used for legal advice or decision-making, emphasizing the need for domain expertise and oversight in applying ai to sensitive legal domains. correctness is the answer correct, given the context? 5 clarity is the answer clear and free of extensive jargon? 6 completeness does the answer fully address all parts and sub-questions? • a user query, your job is to judge the relevance and correctness of the generated answer. output a single score that represents a holistic evaluation. you must return your response in a line with only the score. do not return answers in any other format. on a separate line, provide your reasoning for the score as well. follow these guidelines for scoring: