B rief M e: A Legal NLP Benchmark for Assisting with Legal Briefs (2025)

Abstract: A core part of legal work that has been underexplored in Legal NLP is the writing and editing of legal briefs. This requires not only a thorough understanding of the law of a jurisdiction, from judgments to statutes, but also the ability to make new arguments to try to expand the law in a new direction and make novel and creative arguments that are persuasive to judges. To capture and evaluate these legal skills in language models, we introduce BRIEFME, a new dataset focused on legal briefs. It contains three tasks for language models to assist legal professionals in writing briefs: argument summarization, argument completion, and case retrieval. In this work, we describe the creation of these tasks, analyze them, and show how current models perform. We see that today’s large language models (LLMs) are already quite good at the summarization and guided completion tasks, even beating human-generated headings. Yet, they perform poorly on other tasks in our benchmark: realistic argument completion and retrieving relevant legal cases. We hope this dataset encourages more development in Legal NLP in ways that will specifically aid people in performing legal work.

## introduction legal work involves reading, writing and reasoning about large volumes of text @xcite . for instance, in litigation, attorneys must search and interpret prior case law to construct convincing arguments to stake out a strong position. they present these to a judge in the form of a document called a brief. given their general text processing and generation strengths, large language models (llms) could assist legal professionals with brief drafting. to assess this, we develop a suite of three tasks to test llms' capabilities in supporting brief writing tasks: (1) extreme summarizing of an argument in a brief section, (2) completing a missing argument, and (3) retrieving a missing case citation. most prior work in legal nlp focuses on judicial opinions -documents that present the court's reasoning as a synthesis of various legal considerations, ultimately framing the outcome as the natural conclusion given the legal landscape. we focus instead on briefs, which encapsulate the core of an attorney's work and thought process in advocating for a particular interpretation of the law, especially at the appeals stage. this is more appropriate if the goal is to develop llms that assist attorneys. as seen in figure 1 , briefs are split into multiple sections that constitute sub-arguments about the state of the law as interpreted by that attorney. effective arguments fit together into a cohesive whole that logically walks the judge step-by-step to the attorney's preferred outcome. a model that could assist a legal professional in drafting briefs could offer notable productivity gains. although there is a growing number of legal nlp datasets, none contain legal briefs in significant numbers or explicitly capture the logic of legal argumentation (see §2). to this end, we present briefme, a dataset of supreme court of the united states (scotus) briefs with annotations to support three tasks. we construct briefme from briefs hosted on the supreme court's website. we formulate the tasks based on the structure of scotus briefs. this structure follows a convention where each section and subsection state a concise version of the arguments that are contained in that section, with each section building toward the ultimate argument. see an example in figure 1 . the first task, argument summarization, requires generating a section heading that captures the essence of presented legal arguments. doing so assists attorneys in developing and structuring their briefs. the goal of the second task, argument completion, is to identify and generate a missing argu-figure 1 : we illustrate the typical structure of briefs and highlight specific parts that serve as input and desired output for the argument summarization, argument completion, and case retrieval tasks. ment in a sequence of brief headings. this, in turn, supports attorneys in developing complete and logically coherent arguments, while also serving as a test of models' "understanding" of legal reasoning. finally, case retrieval tests a model's ability to identify relevant precedent that supports specific legal claims. such retrieval ensures that attorneys provide comprehensive citations of prior case law. we develop an llm-as-a-judge using o3-mini instructed based on expert-written guidelines for writing brief headings and tables of contents. although we carefully recruit human judges, their ratings often lack consistency or include justifications irrelevant to the task. thus, to assess the reliability of the llm judge, a qualified author provided meta-ratings of judge performance. we find that the llm-as-judge is a more reliable evaluator than human judges. we also use it to filter out instances with low-quality headings and table of contents in the final version of briefme. we benchmark briefme tasks against a wide variety of current llms and retrieval methods. gpt-4o achieves the strongest performance with average ratings of 4.3 out of 5 for both argument summarization and guided argument completion. in contrast, human-authored headings on the full unfiltered set score lower, averaging 3.4 for summarization and 3.5 for completion, with 43% and 22% of examples with scores of 3 and 3.5 or lower, respectively (fig. 8 in appendix shows the distribution of scores). according to the judge, common issues include lack of persuasive phrasing and weak logical integration with surrounding headings. gpt-4o's outputs, on average, require only minor edits to improve clarity. notably, llms' performance generalizes to briefs published after february 2025. these results suggest llms are wellpositioned to assist in guided argument completion and summarization. future work could study teams of experts and llms assisting with these drafting tasks. however, model performance drops significantly on the other two tasks. gpt-4o identifies the correct location of a missing argument in only 18% of cases. for case retrieval, the best-performing method retrieves the correct precedent among its top five results in just 31.5% of instances. these findings point to the need for further advances before llms can reliably support legal professionals in these more complex drafting scenarios. 1 ## related work processing legal text is an active area of nlp research that has numerous tasks such as legal judgment prediction @xcite @xcite @xcite , legal text classification @xcite @xcite , and multiple-choice qa @xcite @xcite . below, we compare our work to related tasks. legal summarization is an established task in legal nlp with many available resources @xcite @xcite @xcite @xcite . our argument summarization task differs from this prior work by focusing on generating forwardlooking arguments presented by litigants to persuade a court, rather than backward-looking judicial decisions that explain the court's reasoning. this distinction also holds for legal summarization works that aim to summarize other scotus documents, such as casesumm @xcite , which focuses on opinions rather than briefs. our argument completion task does not have a close parallel with another legal nlp task. legal argument mining involves identifying argumentative components in legal texts, such as issues, evidence, and conclusions @xcite , and determining how these components relate to one another. some approaches use richer annotation schemes to better capture the complexity of legal arguments @xcite . however, this task is orthogonal to our argument completion task, which requires generating a missing argument. a broader line of work on ai in debating is related, including ibm's project debater and efforts to improve alignment through debate @xcite . however, these approaches primarily focus on engaging in debates, rather than completing partial argument structures in legal briefs. our case retrieval can be seen as an instance of "case law" retrieval where relevant cases need to be retrieved given a query case, question, or some other text @xcite @xcite @xcite @xcite . the task of predicting entailment relation between two cases is also related @xcite . the distinguishing feature of our case retrieval is that queries are from legal briefs. since briefs are written by lawyers, not judges, their writing style and how they marshal precedent is closer to how most attorneys work. two notable nlp corpora include briefs: @xcite scraped scotus dockets and included briefs in a large pretraining corpus, but the data were not processed for or evaluated on summarization, argument completion, or retrieval tasks. @xcite include briefs in their court filings, but the corpus is smaller, and the briefs are written for trial court motions. ## dataset and task setup we introduce briefme and formulate three tasks designed to assist lawyers in writing briefs: argument summarization, argument completion, and case retrieval. our full data extraction and cleaning pipeline is shown in fig. 3 in the appendix. this was used to collect and clean our raw data to produce our three tasks. ## experiments in this section, we describe the experiments conducted to develop the llm judge and to benchmark models on the briefme tasks. in §4.1 we describe the llm judge, which is a key part of our evaluation protocol. then we describe the experiments and results for argument summarization §4.2, argument completion §4.3, and case retrieval §4.4. ## conclusions we present briefme, a dataset designed to support the development of models that assist in drafting legal briefs. we outline the procedure for collecting this dataset from the web, formulate key tasks, and evaluate strong baselines in fully fine-tuned, few-shot, and zero-shot settings. our findings suggest that briefme could enable models that assist with argument summarization and completion by generating headings that surpass the quality of current human-written ones while also driving much-needed advances in case retrieval.