Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluator (2025)

Abstract: Prompt optimization aims to systematically refine prompts to enhance a language model’s performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.

## introduction terms of service (tos) agreements are lengthy, complex documents that define the legal relationship between companies and consumers. while these documents are critical for protecting consumer rights and regulating corporate practices, clauses in tos agreements are often written in highly complex language, making them difficult for the users to understand. as a result, unfair or potentially exploitative tos clauses, may go unnoticed. detecting such unfair clauses is therefore essential for promoting transparency, consumer protection, and regulatory compliance. manual review of such documents is however extremely time-consuming and requires legal knowledge. large language models (llms) therefore offer a promising alternative by automatically classifying unfair tos clauses at scale. nevertheless, the performance of llms is highly sensitive to the prompt design. prior work has shown that even * these authors contributed equally to this work minor variation in prompt wording and formatting can substantially affect accuracy and consistency @xcite . recently, there is growing research interest in prompt optimization, which is the process of systematically refining prompts to improve a language model's performance on a specific task @xcite @xcite @xcite . prompt optimization is typically framed as an iterative search process that involves modules such as generating revised candidate prompts, evaluating prompt performance, and searching for the best-performing candidates to guide subsequent refinements. despite recent advances, most optimization methods still struggle with inefficient exploration of candidate prompts space. for example, beam search, a widely used search strategy in prior work @xcite , often produces repetitive and untargeted edits, relying on costly deterministic forward-only search. in this work, we adopt monte carlo tree search (mcts) @xcite , inspired by promptagent @xcite to improve the exploration efficiency of the candidate prompts space. mcts strategically models the search space as a tree and updates future reward estimates through backpropagation. another major bottleneck in current prompt optimization methods is the high computational cost of evaluating candidate prompts. each evaluation typically requires costly inference on llms and is repeated across a large pool of candidates. to reduce computation, most methods assess prompt performance using a small score set, a subset sampled from the full validation set. while this approach is faster and cheaper, the small size of the subset can cause performance estimates to fluctuate depending on which samples are included. moreover, prompts optimized on a small score set may fail to generalize well to the full dataset. to mitigate this, we augment our framework with a proxy prompt evaluator based on a correctness classifier, inspired by prompteval @xcite . this proxy scorer efficiently evaluates the performance of prompt candidates by predicting their correctness on the target task, reducing the need for repeatedly calling costly llm and therefore enabling evaluation of the prompts across the full validation set. our prompt optimization framework enables efficient exploration of the prompt search space and streamlines evaluation. our results show that the mcts approach discovers better-performing prompts than existing optimization frameworks and, when combined with a proxy prompt evaluator, achieves similar binary classification performance with reduced computational cost. ## dataset we conduct our prompt optimization experiments on the claudette dataset @xcite , which contains 20,417 clauses extracted from 50 tos contracts of global online platforms. each clause was manually labeled by legal experts as fair or unfair. * a clause is labeled unfair if it somehow introduces an unacceptable imbalance in the parties' rights and obligations, i.e., harms the user's rights or minimizes the online service's obligations. in addition, each unfair clause is annotated with one or more of nine unfairness categories (e.g., arbitration, content removal, jurisdiction) depending on the source of the unfairness. figure 2 exhibits example tos clauses for claudette. notably, the label ratio in claudette is extremely imbalanced, with a distribution of roughly 9:1 (fair:unfair), as displayed in ## methodology we investigate the performance of our prompt optimization framework for the task of unfair tos clause detection. we begin with a simple initial prompt "is this clause fair (0) or unfair (1) to the consumer?". the key modules of our prompt optimization process is illustrated in figure 1 . specifically, we update the prompt and generate various prompt candidates using textual gradients @xcite ( §4.1). to efficiently search among candidate prompts, we employ monte carlo tree search (mcts) following wang et al. 2023 ( §4.2). a major bottleneck of this approach is the high computational cost when evaluating among candidate prompts. to mitigate this, we propose training an external prompt grader model (prompteval (polo et al., 2024)) as a proxy selection module ( §4.3). ## evaluation results in this section, we evaluate the effectiveness of our approach, which integrates mcts with a proxy scorer, on the task of tos fairness classification on the test set. we benchmark our method against three baselines categories: (1) traditional classifiers finetuned on the whole train set (svm, bert), (2) zero-shot llm performance, and (3) baseline prompt optimization methods (opro, grips). for all prompt optimization methods, we report the performance of the final highest-scoring prompts identified by each method. we report both accuracy and macro f1, as the test set is heavily classimbalanced. in addition, we conduct an ablation study to assess the contribution of the proxy scorer. finally, we complement the quantitative results with a qualitative analysis to offer a concrete insights of the improved prompts. ## conclusion in this paper, we propose augmenting a prompt optimization framework with a proxy prompt scorer. our experimental results show that using a lightweight correctness prediction model as a proxy enables existing prompt optimization techniques to avoid repeatedly querying an llm over the validation set, which is computationally expensive. in particular, the mlp-based proxy evaluator achieves performance close to the best-performing standard implementation that relies, while significantly reducing computation time and cost, which highlighting the effectiveness of our methodology. for future work, several promising directions can be explored. first, experimenting alternative proxy model architectures, including transformers or bayesian models to better capture the interaction between prompt and task performance. another promising direction is to optimize the score set us-ing active or curriculum learning strategies. by selectively including the most informative or representative clauses, active learning can reduce the number of evaluations required while maintaining reliable performance estimates. similarly, curriculum learning can improve the proxy model's stability by starting with easier examples and progressively incorporating harder ones. these approaches would allow for more efficient and effective prompt evaluation, reducing computational cost while improving generalization to the full dataset.