Evaluating the Role of Verifiers in Test-Time Scaling for Legal Reasoning Tasks (2025)

Abstract: Test-time scaling (TTS) techniques can improve the performance of large language models (LLMs) at the expense of additional computation and latency. While TTS has proven effective in formal domains such as mathematics and programming (Snell et al., 2024; Chen et al., 2024), its value in argumentative domains such as law remains underexplored. We present an empirical study of verifier-based TTS methods for legal multiple-choice QA (MCQA) across five benchmarks. Using a family of 7 reward models, we evaluate both outcome-level (Best-of- N ) and process-level (tree search) verification under realistic low- N budgets. Our analysis systematically investigates how verifier utility is affected by key properties such as domain specialization, model size, and supervision type (process-supervised PRMs vs. outcome-only ORMs), even when applied across different roles.

## introduction test-time scaling (tts) methods aim to enhance large language model (llm) performance by trading additional compute for improved accuracy at inference time. the broad spectrum of these techniques range from single-path approaches like generating longer chains-of-thought (cot) @xcite @xcite to more complex parallel and verifier-guided methods such as best-of-n (bon) selection and tree search. systematic investigations of these verifier-guided methods in formal domains like math and programming have demonstrated substantial accuracy improvements on multiple choice qa (mcqa) tasks @xcite @xcite . however, the legal domain presents distinct challenges; its reasoning is often defeasible and accommodates multiple valid analytical paths. while prior work has explored single-path inference for legal reasoning @xcite , investigations into parallel and verifier-based tts are notably absent from the literature. this gap is critical because the verifiers underpinning these methods are often trained on general-purpose or formal-domain data. it remains an open question whether such models can reliably evaluate legal reasoning, or if domain-specific verifiers are required to achieve meaningful gains. these verifiers are broadly categorized into two types: outcome reward models (orms), which assign a single score to a complete output, and process reward models (prms), which provide fine-grained, step-by-step feedback (uesato et al., 2022; lightman et al., 2023). this paper addresses the aforementioned gap by empirically investigating whether the performance enhancements observed in formal domains translate to legal mcqa. through an extensive comparison of reward models, we analyze how to optimize these verification strategies by evaluating the impact of verifier domain specialization, model size, and supervision type (prm vs. orm). we examine verifier-based tts for legal reasoning to answer the following research questions: • rq2 (importance of domain specialization and verifier size). with method and compute held constant, does a legal-specialized verifier outperform a general-domain verifier, and how large is the additional effect of scaling the verifier size? contributions this paper makes three key contributions to the understanding of verifier-based test-time scaling for legal reasoning. first, we conduct a comprehensive comparison between mv, bon, and process-verified dvts using opensource models, revealing that verifier-based methods rarely outperform simple voting baselines by significant margins in legal reasoning. second, through systematic ablation studies, we show that both verifier model size and domain specialization are crucial for improving performance, with legaldomain training providing a distinct advantage that becomes most apparent at larger scales. notably, we find that the utility of all methods diminishes as generator model capability increases, with even sophisticated verification providing minimal gains. finally, our analysis of supervision type shows that prms deliver superior performance compared to orms of similar size, even when prms are deployed outside their intended role for outcome verification tasks. these findings provide valuable guidance for practitioners seeking to optimize computational resources in legal nlp applications. ## experimental setup we test three generators: llama-3.2-3b-instruct, llama-3.1-8b-instruct, and llama-3.1-70b-instruct @xcite . we ran our tests with cot prompting @xcite and temperature @xmath0.8 and the system prompts in appendix b. our evaluation compares three methods: majority vote (mv): sample k cot responses and select the most frequent answer. best-of-n (bon): sample n cot responses, score each with an outcome reward model (orm), and select the one with highest reward. dvts @xcite : a tree search guided by a process reward model (prm) scoring partial steps. for optimal hyperparameter choice, such as expansion width or aggregation strategy, we ran an ablation study on mbe exam that can be found in appendix c. while generating n trajectories of length t dominates computational cost at o(t 2 ), the reward-model scoring methods (best-of-n and dvts) add only modest linear o(t ) overhead, making all three approaches comparable in runtime (see more details in appendix a) . we use the verifiers detailed in type verifier model size training data orm our rms 8b, 70b general + legal † skywork-reward (liu et al., 2024) 8b, 27b general prm versaprm (zeng et al., 2025) 8b multi-domain * qwen2.5-math-prm (zhang et al., 2025) 7b, 72b math ## conclusions in this work, we presented a systematic evaluation of verifier-based tts for legal multiple-choice question answering. a consistent observation is the diminishing return of verification as the generator model's power increases; while gains are evident for smaller generators, they shrink significantly for more powerful ones, where a simple majority vote often remains competitive. however, the utility of majority vote is challenged by task complexity, and we find that verifier-based methods provide substantial gains in high-cardinality benchmarks where the answer space is large. crucially, we find that effective verification relies on the dual impact of model scale and domain specialization. the advantage of in-domain training is most pronounced at larger verifier scales. this is complemented by the notable generalization of process supervision: versaprm proved highly versatile, outperforming size-matched outcome reward models even when used out-of-role for outcome reranking. for practitioners, these findings suggest that investing in high-quality, in-domain reward models is a promising direction for improving inference-time legal reasoning.