G reek B ar B ench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations (2025)

Abstract: We introduce GreekBarBench, a benchmark that evaluates LLMs on legal questions across five different legal areas from the Greek Bar exams, requiring citations to statutory articles and case facts. To tackle the challenges of free-text evaluation, we propose a three-dimensional scoring system combined with an LLM-as-a-judge approach. We also develop a meta-evaluation benchmark to assess the correlation between LLM-judges and human expert evaluations, revealing that simple, span-based rubrics improve their alignment. Our extensive evaluation of 13 proprietary and open-weight LLMs shows that even though the top models exhibit impressive performance, they remain susceptible to critical errors, most notably a failure to identify the correct statutory articles.

## introduction as legal ai assistants become increasingly prevalent, the need for realistic legal llm benchmarks has never been more imperative. 1 most widely used legal natural language processing (nlp) benchmarks @xcite focus on classification tasks, e.g., legal judgement prediction @xcite , which have been criticized @xcite for being more constrained and less representative than real-world tasks. even more recent llm-focused legal benchmarks @xcite @xcite do not go beyond closed-form questions (e.g., multiple-choice questions), failing to capture the true complexity of legal reasoning in practice, which involves identifying, analyzing and synthesizing relevant information to reach a conclusion. unfortunately, most existing benchmarks facts [1] antonis visited his dermatologist, ioannis, to remove facial skin tags. ## automatic evaluation to address the evaluation of free-text answers without the prohibitive cost of manual annotations, we use the llm-as-a-judge framework @xcite . llm-judges can be categorized into two primary types: (a) pairwise llm-judges, which evaluate two candidate answers and determine which is preferred (or declare a tie), and (b) grading llmjudges, which assign an integer score to each individual candidate answer @xcite . in our work, we focus on grading llm-judges to allow cost-effective integration of new participant llms without the overhead of quadratically increasing pairwise comparisons. to improve the alignment of llm-judges with human expert annotators, we propose novel spanbased rubrics; i.e., evaluator instructions in the form of annotated spans per question. these spans will guide the llm-judge in what to assess in the candidate answers. however, even with these question-specific rubrics, replicating the nuanced evaluation of human experts, especially in complex tasks like legal writing, cannot be guaranteed. for this reason, we also include a framework to metaevaluate whether llm-judges are suitable proxies for human evaluation on greekbarbench. ## meta evaluation meta-evaluation of grading llm-judges aims to quantify the alignment between llm-generated scores and human expert annotations. previous research has predominantly relied on pearson's or spearman's correlation coefficients as primary meta-metrics @xcite , often without substantial justification. however, advancements in meta-evaluation have emerged from the machine translation domain, particularly through the wmt metrics shared task @xcite , where automatic evaluation frameworks have been systematically compared and refined. the task aims to identify optimal metrics for translation quality assessment by com-paring system outputs against references. recent findings demonstrate that state-of-the-art metrics are increasingly llm-based. the task has revealed that pearson's correlation coefficient exhibits vulnerability to outliers, while spearman's ρ disregards the magnitude of ranking errors, applying uniform penalties. to address these limitations, wmt has adopted soft pairwise accuracy (spa) @xcite , a metric that assigns partial credit for nearly correct rankings, thereby providing an evaluation framework that better reflects the alignment of metrics with human experts. ## manual error analysis to supplement our quantitative metrics, we conducted a manual error analysis to identify common failure modes. we analyzed the 10 worst-scoring responses from five key models. human experts, guided by the scores and explanations of the llmjudge, categorized the errors to reveal the most significant weaknesses. we focused on critical errors that led to an incorrect answer. the types of errors are the following: facts: errors include hallucinations ('hall.'), inventing facts not present in the provided context, and inaccuracies ('inac.'), misinterpreting or omitting key facts from the context. cited articles: errors include inapplicable articles ('inap.'), citing legal articles not relevant to the case, and missing articles ('missing'), failing to identify essential articles for the analysis. analysis: errors include an invalid argument ('argum.'), a flawed argument from misunderstood principles or fallacious reasoning, and a generic answer ('generic'), providing a general statement instead of an analysis tailored to the case facts. our analysis, while based on a small sample, highlights three primary areas of weakness: citing legal articles, understanding case facts, and constructing valid arguments. the most frequent errors related to failing to cite the correct legal articles (23). this supports our finding ( §5.5) that correct article retrieval is essential. factual (20) and analysis (17) errors were also prevalent across all models. although rare, critical fact hallucinations (3) appeared even in top-performing models, highlighting the risks of deploying these systems without human oversight. a key differentiator between human and model performance was the inability of llms to grasp ## conclusions in this work, we introduced greekbarbench, a benchmark evaluating llms on legal questions requiring citations to statutory articles and case facts. to enable robust and scalable assessment, we developed a comprehensive evaluation framework based on llm-as-a-judge, which we validated against human experts using our gbb-jme metaevaluation benchmark. our results show that custom span-based rubrics significantly improve judge alignment. the extensive evaluation of 13 llms on greekbarbench revealed that gemini-2.5-flash and gpt-4.1 achieved the best performance. our primary finding is that success on this task is critically dependent on the quality of the provided legal context and the ability to identify the correct statutory articles within it.