L e MAJ (Legal LLM -as-a-Judge): Bridging Legal Reasoning and LLM Evaluation (2025)

Abstract: Evaluating large language model (LLM) outputs in the legal domain presents unique challenges due to the complex and nuanced nature of legal analysis. Current evaluation approaches either depend on reference data, which is costly to produce, or use standardized assessment methods, both of which have significant limitations for legal applications.Although LLM-as-a-Judge has emerged as a promising evaluation technique, its reliability and effectiveness in legal contexts depend heavily on evaluation processes unique to the legal industry and how trustworthy the evaluation appears to the human legal expert. This is where existing evaluation methods currently fail and exhibit considerable variability.This paper aims to close the gap: a) we break down lengthy responses into “Legal Data Points” (LDPs) — self-contained units of information — and introduce a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers; b) we demonstrate that our method outperforms a variety of baselines on both our proprietary dataset and an open-source dataset (LegalBench); c) we show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement; and finally d) we open source our Legal Data Points for a subset of LegalBench used in our experiments, allowing the research community to replicate our results and advance research in this vital area of LLM evaluation on legal question-answering.

## introduction large language models (llms) are increasingly integrated into legal workflows, supporting tasks such as contract analysis @xcite , document classification @xcite , information extraction @xcite and court case outcome prediction @xcite . lawyers and legal professionals are relying more and more on legal outputs generated by an llm to counter increasing pressure to 'do more for less' and this shows no sign of slowing down as the quality of legal ai tools increases @xcite . among these applications, question-answering over legal documents has emerged as a particularly valuable task. legal evaluation (commonly referred to as 'legal review' in legal circles) is a crucial component of legal tasks, as it ensures that legal outputs are correct, complete and relevant. however, there are many challenges plaguing the evaluation of such outputs; among other things, it is time-consuming, requires highly specialized, expensive human experts and is prone to subjectivity @xcite @xcite . traditional automated evaluation methods, which rely on high-quality reference answers ('ground truths') @xcite , have partially mitigated these challenges. yet, their reliance on multiple expert lawyers to construct comprehensive ground truths @xcite significantly limits their scalability, particularly in the legal domain, where the nuance and variability of legal language amplifies the requirements for large, representative datasets for reliable real-world evaluation. llm-as-a-judge has emerged as a promising evaluation paradigm that can assess outputs without reference data and provide explanatory feedback @xcite . however, these techniques can rely on hard to verify assessment approaches and can be prone to bias and task variability @xcite . existing questionanswering methods that assess the whole answer as a unit @xcite fail to capture the granular assessment process that legal professionals employ @xcite . recent research @xcite has also shown that llm-as-a-judge methods perform significantly worse without good quality references. our research reveals a critical insight: automated evaluations correlate significantly better with human expert judgments, even without human references, when they mirror how lawyers actually evaluate answers. from interviews with legal experts, we understand that there is added complexity and logic to legal reasoning that needs to be built into the llm-as-a-judge paradigm, bridging the gap between both methods for better alignment. we propose lemaj (legal llm-as-a-judge), a novel evaluation methodology specifically designed to emulate lawyers' evaluation processes. our approach decomposes llm-generated answers into discrete "legal data points"-selfcontained units of information-and systematically evaluates each for correctness and relevance while identifying critical omissions. this granular assessment provides the detailed feedback legal practitioners require beyond simple accuracy scores, drawing inspiration both from techniques in summary evaluation @xcite and from our own user study. through empirical studies, we first demonstrate that using ldps improves alignment between human and llm evaluation and correlation with goldstandard meta-reviews. we will present how our lemaj framework consists of two core elements: the lemaj automated evaluation based on ldp segmentation resulting in correctness and relevance scores, as well as a user interface, displaying ldps for annotation by human legal experts. we then show how our lemaj automated evaluation substantially outperforms various llm and non-llm baselines when evaluated on both legal-bench @xcite )-an open-source le-gal dataset-and proprietary legal data. next, we show how the lemaj user interface component improved inter-annotator agreement between human legal experts when reviewing with lemaj. finally, we introduce a commercial use case for the use of lemaj by showing time savings through the triaging of answers for review. to sum up, the main contributions of this article are the following: ## related work automatic evaluation: there are a number of evaluation methods used in the nlp space -n-gram based methods such as rouge @xcite and bleu @xcite , or token level modelbased methods such as bertscore @xcite or bartscore @xcite . however, these methods require a human reference. we also look at llm-as-a-judge methods, such as deepeval @xcite , but while they can be used without references, they tend to perform poorly when used in a reference-free setting, especially when the llm itself is not able to produce a correct answer @xcite . furthermore, model-based methods like llm-as-a-judge suffer from task variability @xcite , meaning that one solution to fit all tasks is unlikely to exist and people often tend to customize any existing offering. this calls for adapting existing llm-as-a-judge methods to legal specific tasks such as legal q&a. replicating the reasoning process: automated evaluation methods are usually evaluated by computing a correlation against human scores @xcite . given the reliance on correlation with humans, it becomes imperative that higher levels of inter-annotator agreement are reached. there are a few factors that can contribute to rating disagreements such as problems with the study setup, insufficient information provided on the rating scale, complex or highly subjective tasks, or even the annotator's background @xcite . as a result, being able to reduce these factors has become a big research area, especially in situations where the answers being evaluated are verbose. work in this space has proposed dissecting the answer into easier to evaluate units @xcite , splitting the evaluation pipeline into interpretable measures @xcite , or in cases where a high-quality reference output is available, measuring the similarity between the two @xcite . but issues still persist as the actual measure is dependent on the task at hand, with some favoring factuality and reducing hallucinations @xcite , while others focus on correctness and human alignment. therefore we identify a need for a flexible and easily adaptable metric. ## conclusion we introduced lemaj, a novel evaluation framework that seeks to closely emulate the legal reasoning process. we have shown that by splitting up a legal answer into single units of information (legal data points) we can use llms to evaluate the correctness and relevance of a given answer at more granular level. the results of this methodology show a stronger correlation between the lemaj evaluation and a human gold standard than existing benchmarks on both our proprietary dataset and a subset of an open-source dataset, legalbench. additionally, we demonstrate that lemaj can improve inter-annotator agreement on correctness, addressing a critical issue in the development of high-quality reference data and for the effective evaluation of assessment methods. finally, we showcase the time savings in a practical application and a potential deployment pathway for lemaj. future work will look at increasing the accuracy of the method, improving its ability to detect incorrect and missing information even further, as well as extending the scalability work to a multi-agent framework that can detect the needs of a task and adapt the metric on the fly, attempting to improve issues around task variability in a single llm-as-a-judge framework.