PILOT -Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC -Aligned Classification Tasks (2025)

Abstract: The Patent Trial and Appeal Board (PTAB) of the USPTO adjudicates thousands of *ex parte* appeals each year, requiring the integration of technical understanding and legal reasoning. While large language models (LLMs) are increasingly applied in patent and legal practice, their use has remained limited to lightweight tasks, with no established means of systematically evaluating their capacity for structured legal reasoning in the patent domain. To address this gap, we introduce **PILOT-Bench** (**P**atent **I**nva**L**idati**O**n **T**rial Benchmark), a dataset and benchmark that aligns PTAB decisions with USPTO patent data at the case level and formalizes three IRAC-aligned classification tasks: Issue Type, Board Authorities, and Subdecision. We evaluate a diverse set of commercial and open-source LLMs and conduct analyses across multiple perspectives, including input-variation settings, model families, and error tendencies. Notably, on the Issue Type task, commercial models consistently exceed 0.75 in Exact Match, whereas the strongest open-source model (Qwen-8B) achieves performance around 0.56, highlighting the substantial gap in reasoning capabilities. PILOT-Bench establishes a foundation for the systematic evaluation of patent-domain legal reasoning and points toward future directions for improving LLMs through dataset design and model alignment. All data, code, and benchmark resources are available at https://github.com/TeamLab/pilot-bench.

## introduction as the volume of patent applications and examinations continues to grow, the patent trial and appeal board (ptab) of the us patent and trademark office (uspto) handles a substantial number of appeals and invalidation proceedings each * equal contribution. in this paper, we propose the patent invalidation trial benchmark (pilot-bench), a dataset and benchmark for evaluating the legal reasoning abilities of llms in the patent domain. we combine ptab decisions with uspto data per case and construct classification tasks aligned with the issue-rule-application-conclusion (irac) framework commonly used in legal practice. our contributions are threefold: pilot-bench is, to our knowledge, the first benchmark that integrates 18k ptab ex parte appeals with uspto patent text at the case-level and provides 15k opinion-split instances explicitly engineered to prevent label leakage. pilot-bench establishes a benchmark for evaluating llms' legal reasoning in the patent domain-specifically, ptab ex parte appeals where technical understanding and legal reasoning meet. our objective is to open a durable, reusable point of comparison that can anchor subsequent model, data, and methodology work and, ultimately, support responsible use of llms in patent practice. accordingly, we fix the evidence boundary via the opinion split: inputs contain only appellant_arguments and examiner_findings, with all ptab_opinion text excluded. we keep the label schema fixed across issue type, board authorities, and subdecision (fine/coarse) and evaluate under a unified zero-shot protocol with taskappropriate metrics (exact match/macro-f1/micro-f1 for multi-label; accuracy/macro-f1/weighted-f1 for multi-class). we also report results for both closed-source @xcite and open-source model families and for the split (base), merge, and split+claim input-variation settings, providing reference baselines for subsequent work. ## data construction this section describes the construction of the pilot-bench dataset, including source collection, case-level alignment, text normalization, opinion splitting, and label refinement. the goals are threefold: (i) to consistently align ptab decisions with uspto patent information; (ii) to prevent answer leakage by excluding the board's opinion from inputs via the opinion split; and (iii) to provide input-label sets that reflect ptab practice and are directly applicable to irac-aligned classification tasks. ## tasks in this section, we formalize the benchmark's three classification tasks in alignment with the irac framework. while we follow irac's logical order, the tasks are defined as independent evaluation units without dependencies across them. a uniform input and leakage-prevention policy applies: to avoid answer leakage, we exclude all ptab's opinion text, and by default inputs consist only of the appellant_arguments and examiner_findings produced by the opinion split. we note that the benchmark does not include a task corresponding to the application stage of irac. application requires multi-step reasoning that connects legal rules to case-specific facts, which goes beyond the scope of classification. in this work, we focus on classification tasks as a first step, and leave application to future research, where it can be more appropriately modeled through generation tasks that capture complex legal reasoning. ## experiments we describe the experimental setup, model lineup, and evaluation protocol for the three classification tasks. unless otherwise noted, inputs are restricted to the appellant_arguments and examiner_findings obtained from the opinion split, with all ptab's opinion text excluded. for input-variation experiments, we compare three configurations under identical instructions: split (base), merge, and split+claim. in the split (base) setting, appellant and examiner arguments are separated into distinct segments. merge combines the two roles into a single role-neutral input, while split+claim augments the role-separated arguments with the patent's claim text. these variants allow us to analyze the relative contributions of role signals (the distinction between appellant and examiner) and technical signals (the claim text) to model performance. the model lineup includes five closedsource(commercial) llms and four open-source llms. the closed-source(commercial) models are claude-sonnet-4 (anthropic, 2025), gemini-2.5-pro (gemini team, 2025), gpt-4o, gpt-o3 (openai, 2024), and solar-pro2 @xcite . the open-source models are llama-3.1 (meta ai, 2024), mistral @xcite , qwen (qwen team, 2025), and t5 (google deepmind, 2025). in the instruction and applying post-processing to convert outputs into valid json. this ensured that parsing errors were minimized across all runs. all tasks are evaluated in a zero-shot setting under a unified protocol. detailed instruction templates, and prompts are provided in appendix d.3 and model specifications are provided in the appendix f . ## results we evaluate model performance across the three classification tasks, with task-level results reported in tables 2a-3b ; confusion heatmaps appear in the appendix e.2. overall, closed-source(commercial) models consistently outperform open-source models, although all models exhibit limitations under long-tailed label distributions. macro-f1 remains low across tasks, reflecting persistent difficulty with rare labels. ## conclusion we presented pilot-bench, a benchmark to evaluate legal reasoning in the patent domain by aligning ptab ex parte appeals with uspto patent data. by framing three irac-aligned classification tasks, we enable systematic assessment of llms' ability to identify issues, map rules, and predict conclusions in appeal proceedings. our experiments show that while closed-source(commercial) llms outperform open-source models, all models face persistent challenges with label imbalance and procedural-rule mapping. input-variation analysis further demonstrates that simply adding all claims can harm performance, underscoring the need for more targeted data design. pilot-bench thus provides both a resource and an evaluation protocol to study how llms reason in a domain where technical detail and legal precision must be combined. we hope this benchmark will encourage further work at the intersection of nlp, law, and intellectual property.