H y PA - RAG : A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications (2025)

Abstract: Large Language Models (LLMs) face limitations in AI legal and policy applications due to outdated knowledge, hallucinations, and poor reasoning in complex contexts. Retrieval-Augmented Generation (RAG) systems address these issues by incorporating external knowledge, but suffer from retrieval errors, ineffective context integration, and high operational costs. This paper presents the Hybrid Parameter-Adaptive RAG (HyPA-RAG) system, designed for the AI legal domain, with NYC Local Law 144 (LL144) as the test case. HyPA-RAG integrates a query complexity classifier for adaptive parameter tuning, a hybrid retrieval approach combining dense, sparse, and knowledge graph methods, and a comprehensive evaluation framework with tailored question types and metrics. Testing on LL144 demonstrates that HyPA-RAG enhances retrieval accuracy, response fidelity, and contextual precision, offering a robust and adaptable solution for high-stakes legal and policy applications.

## introduction large language models (llms) like gpt @xcite , gemini @xcite , and llama @xcite have advanced question answering across domains @xcite @xcite . however, they face challenges in domains like law and policy due to outdated knowledge limited to pre-training data @xcite and hallucinations, where outputs appear plausible but are factually incorrect @xcite . empirical evidence indicates that many ai tools for legal applications overstate their ability to prevent hallucinations @xcite . cases of lawyers penalized for using hallucinated court documents @xcite highlight the need for reliable ai systems in legal and policy contexts. ## background and related work recent llm advancements have influenced law and policy, where complex language and large text volumes are common @xcite @xcite . llms have been applied to legal judgment prediction, document drafting, and contract analysis, improving efficiency and accuracy @xcite @xcite . techniques like fine-tuning, retrieval augmentation, prompt engineering, and agentic methods have further enhanced performance in summarization, drafting, and interpretation @xcite . rag enhances language models by integrating external knowledge through indexing, retrieval, and generation, using sparse (e.g., bm25) and dense (e.g., vector) techniques with neural embeddings to improve response specificity, accuracy, and grounding @xcite @xcite @xcite . to overcome naive rag's limitations, such as poor context and retrieval errors, advanced methods like hybrid retrieval, query rewriters, and rerankers have been developed @xcite @xcite . hybrid retrieval combines bm25 with semantic embeddings for better keyword matching and contextual understanding @xcite @xcite , while knowledge graph retrieval and composed retrievers improve accuracy and comprehensiveness @xcite @xcite . recently, rag systems have advanced from basic retrieval to dynamic methods involving multi-source integration and domain adaptation @xcite . innovations like self-rag and kg-rag improve response quality and minimize hallucinations through adaptive retrieval and knowledge graphs @xcite . frameworks for evaluating rag systems include ragas, which uses reference-free metrics like faithfulness and relevancy (shahul et al., 2023b), giskard, which leverages synthetic qa datasets @xcite , and ares, which employs prediction-powered inference with llm judges for precise evaluation @xcite . ## system design the hybrid parameter-adaptive rag (hypa-rag) system, shown in figure 1 , integrates vector-based text chunks and a knowledge graph of entities and relationships to improve retrieval accuracy. it employs a hybrid retrieval process that combines sparse (bm25) and dense (vector) methods to retrieve an initial top-𝑘 set of results, refined using reciprocal rank fusion based on predefined parameter mappings. a knowledge graph (kg) retriever dynamically adjusts retrieval depth and keyword selection based on query complexity, retrieving relevant triplets. results are combined with the kg results appending it to the retrieved chunks to generate an final set of 𝑘 chunks. optional components include a query rewriter to enhance retrieval with reformulated queries and a reranker for further refining chunk ranking. de-duplicated rewritten query results are integrated into the final set, which, along with knowledge graph triplets, is processed within the llm's context window for precise, contextually relevant responses. the framework has two variations: parameter-adaptive (pa) rag, which excludes knowledge graph retrieval, and hybrid parameter-adaptive (hypa) rag, which incorporates it. ## ai legal and policy corpus local law 144 (ll144) of 2021, enacted by new york city's department of consumer and worker protection (dcwp), regulates automated employment decision tools (aedts). this study uses a 15-page version of ll144, combining the original law with dcwp enforcement rules. as an early ai-specific law, ll144 is included in gpt-4 and gpt-4o training data, verified via manual prompting, and serves as a baseline in this research. the complexity of ll144 motivates our system's design for several reasons: (1) it requires multi-step reasoning and concept linking due to its mix of qualitative and quantitative requirements-definitions, procedural guidelines, and compliance metrics-that semantic similarity alone cannot capture, addressed through our knowledge graph; (2) seemingly simple queries can be ambiguous or require multiple information chunks, making a query rewriter and classifier necessary; and (3) while not specific to our adaptive classifier, the evolving nature of ai laws limits the effectiveness of static pre-training, making retrieval-augmented systems better suited to handle frequent updates. these factors go beyond what standard llms and basic rag systems can manage, justifying the need for our approach. ## performance evaluation the evaluation process starts by generating custom questions tailored to ai policy and legal questionanswering, then introduces and verifies evaluation metrics (see evaluation section of figure 5 in appendix a.2). for reproducibility, the llm temperature is set to zero for consistent responses and all other parameters are set to defaults. ## chunking method we evaluate three chunking techniques: sentencelevel, semantic, and pattern-based chunking. sentence-level chunking splits text at sentence boundaries, adhering to token limits and overlap constraints. semantic chunking uses cosine similarity to set a dissimilarity threshold for splitting and includes a buffer size to define the minimum number of sentences before a split. pattern-based chunking employs a custom delimiter based on text structure; for ll144, this is "\n §". figure 3 shows that pattern-based chunking achieves the highest context recall (0.9046), faithfulness (0.8430), answer similarity (0.8621), and correctness (0.7918) scores. sentence-level chunking, however, yields the highest context precision and f1 scores. semantic chunking performs reasonably well with increased buffer size but generally underperforms compared to the simpler methods. further hyperparameter tuning may improve its effectiveness. these findings suggest that a corpusspecific delimiter can enhance performance over standard chunking methods. for subsequent experiments, we adopt sentencelevel chunking with a default chunk size of 512 tokens and an overlap of 200 tokens. ## query complexity classifier we developed a domain-specific query complexity classifier for adaptive parameter selection, mapping queries to specific hyper-parameters. unlike adaptive rag @xcite , our classifier influences not only the top-𝑘 but also knowledge graph and query rewriter parameters. our analysis of top-𝑘 selection indicated different optimal top-𝑘 values for various question types, as shown in figure 6 (appendix a.4). ## overall results and discussion our analysis demonstrates that adaptive methods outperform fixed baselines, particularly in faithfulness and answer quality. adaptive parameters, such as query rewrites and reranking, enhance response accuracy and relevance, though reranking may slightly reduce overall correctness scores, indicating a trade-off between precision and quality. adding a knowledge graph maintains correctness but introduces complexity, potentially lowering response quality. however, combining adaptive parameters with reranking maximizes correct responses, even if it doesn't achieve the highest scores across all metrics. these findings demonstrate the effect of adaptivity and parameter tuning to balance performance, enabling effective handling of diverse and complex queries. this suggests our system could also apply to other domains where queries demand complex, multi-step reasoning and non-obvious concept relationships. limitations and future work are detailed in appendix a.13. ## ethical considerations the deployment of the hybrid parameter-adaptive rag (hypa-rag) system in ai legal and policy contexts raises critical ethical and societal concerns, particularly regarding the accuracy, reliability, and potential misinterpretation of ai-generated responses. the high-stakes nature of legal information means inaccuracies could have significant consequences, highlighting the necessity for careful evaluation. we emphasize transparency and reproducibility, providing detailed documentation of data generation, retrieval methods, and evaluation metrics to facilitate replication and scrutiny. the environmental impact of nlp models is also a concern. our system employs adaptive retrieval strategies to optimize computational efficiency, reduce energy consumption, and minimize carbon footprint, promoting sustainable ai development. our findings enhance the understanding of rag systems in legal contexts but are intended for research purposes only. hypa-rag outputs should not be used for legal advice or decision-making, emphasizing the need for domain expertise and oversight in applying ai to sensitive legal domains. 4 evaluation results for varied top-𝑘 figure 6: rag evaluation metrics for varied top-𝑘 a.5 human annotation criteria no. criterion description 1 faithfulness are all claims in the answer inferred from the context? 2 answer relevancy is the answer relevant to the question? 3 context relevancy is the context relevant to the question? 4