Are LLM s Court-Ready? Evaluating Frontier Models on I ndian Legal Reasoning (2025)

Abstract: Large language models (LLMs) are moving into legal workflows, yet we lack a jurisdiction-grounded way to gauge their basic competence in thereof. We use India’s public legal examinations as a transparent proxy. Our multi-year benchmark assembles objective screens from top national and state exams and evaluates open and frontier LLMs under real world exam conditions. To probe beyond MCQs, we also include a lawyer-graded, paired-blinded study of long-form answers from the Supreme Court’s Advocate-on-Record exam. This is, to our knowledge, the first exam-grounded, India-specific yardstick for LLM court-readiness released with datasets and protocols. Our work shows that while frontier systems consistently clear historical cutoffs and often match or exceed recent top-scorer bands on objective exams, none surpasses the human topper on long-form reasoning. Grader notes converge on three reliability failure modes—procedural/format compliance, authority/citation discipline, and forum-appropriate voice/structure. These findings delineate where LLMs can assist (checks, cross-statute consistency, statute and precedent lookups) and where human leadership remains essential: forum-specific drafting and filing, procedural and relief strategy, reconciling authorities and exceptions, and ethical, accountable judgment.

## introduction llms have cleared multiple-choice bar-style screens in several jurisdictions, renewing interest in legal automation, but a jurisdiction-first question remains: are these systems court ready? other fields probe such capability with exam-style settings: broad knowledge suites such as mmlu, olympiad-level problems in mathematics and science, and clinically oriented reasoning in health @xcite @xcite . by contrast, many ai-and-law studies focus on short-context recall (for example, bail or recidivism prediction and legal-judgment prediction). these are metric-friendly but only indirectly tied to how courts expect lawyers to write and file @xcite @xcite . india offers a jurisdiction where court-legible benchmarks already exist. we adopt public exams already used to gate human entry common law admission test (clat) -ug/pg (admissions), delhi judicial services/ delhi higher judicial services (djs/d-hjs) prelims (judiciary), and the supreme court's advocate-on-record (aor) exam (rights of audience) as court-ready yardsticks (consortium of national law universities, 2025b; consortium of nlus, 2025; high court of delhi, 2023, 2024; supreme court of india, 2025). our primary contributions in this paper are: we release the dataset here. by anchoring evaluation in public exams that every law student, judge, and policymaker recognizes, we present results that legal practitioners can interpret and ml researchers can reproduce. we hope this shared yardstick helps both communities see where current models stand today and guides evidencebased adoption and research. ## related work legal nlp suites such as lexglue and legal-bench cover broad tasks, and il-tur targets indian legal texts, but none align with the public exams that govern entry and practice in india @xcite @xcite . outside law, exam-style evaluations (e.g., mmlu and olympiad-level benchmarks) stress reasoning but do not test forum-specific procedure or authority discipline @xcite @xcite . in ai and law, much of the literature concentrates on economically salient prediction tasks (for example, bail, recidivism, and legal-judgment prediction), where metrics are tractable but only weakly aligned with the linguistic and rhetorical demands of courtroom writing @xcite @xcite . bar-exam studies are informative but jurisdictionally distinct from india's public exams @xcite . to our knowledge, this is the first study to evaluate frontier llms on india's public legal examinations, pairing multi-year objective screens with a lawyergraded subjective study under exam constraints. ## exams and scope why these exams? india's legal profession is structured around a publicly administered system of entry and advancement that is mediated through a series of high-stakes examinations. these include: (i) the clat-ug/pg for entry into undergraduate and postgraduate law programs, (ii) judicial service examinations such as the djs and dhjs for recruitment to the judiciary, and (iii) the advocate-on-record (aor) examination, which confers exclusive rights of audience before the supreme court of india. reliance on these standardized assessments provides institutional legibility for stakeholders, who use them to regulate access, allocate professional privileges, and validate competence across the legal system for humans. (consortium of national law universities, 2025b; consortium of nlus, 2025; high court of delhi, 2023, 2024; supreme court of india, 2025). ## model selection and inference configuration objective-exam cohort. we evaluate a broad panel spanning frontier proprietary, large open, and exam mod. qs/exam marking total questions clat ug obj. 200/150/120 +1/ -0.25/0 3,154 clat pg obj. 120 +1/ -0.25/0 814 djs (prelim.) obj. 200 +1/ -0.25 1,400 dhjs (prelim.) obj. 150 +1/ -0.25 850 aor 2023 (sc) subj. 35 none (perquestion marks) 35 ## conclusion this study demonstrates a clear dichotomy in ai capabilities for legal tasks. on objective, multiplechoice examinations, frontier models meet or even surpass historical human pass marks, demonstrating a robust capacity for short-context legal recall and rule application. however, this proficiency does not translate seamlessly to the domain of subjective, long-form writing, where no model could match the performance of a human topper. the critical deficits lie both in knowledge and in execution: a lack of procedural fidelity, imprecise authority handling, and a failure to adopt forumspecific structure. these findings compel a two-part definition of what it means to be truly "court-ready." first is the capacity to recall and apply legal doctrine at scale, a benchmark the strongest systems now meet. the second, and more elusive, aspect is reliability under the practical constraints of legal practice, specifically adhering to procedural defaults, maintaining strict authority discipline, and producing work products that align with judicial expectations. our results indicate that current systems fall short on this second, crucial dimension for apex court practice in india. in practical terms, this means ai is best deployed as a supportive tool rather than an autonomous practitioner. systems can efficiently assist with tasks like searching and verifying authorities, checking consistency across drafts, or cross-referencing case details. tasks that require full drafting, independent citation, or any action carrying legal responsibility remain beyond current capabilities and should be handled with human oversight.