PMPO : Probabilistic Metric Prompt Optimization for Small and Large Language Models (2025)

Abstract: Prompt optimization is a practical and widely applicable alternative to fine tuning for improving large language model performance. Yet many existing methods evaluate candidate prompts by sampling full outputs, often coupled with self critique or human annotated preferences, which limits scalability, especially for smaller models or models that are not instruction tuned. We present PMPO (Probabilistic Metric Prompt Optimization), a unified framework that uses token level cross entropy as a direct, lightweight evaluation signal. PMPO locates low quality prompt segments via a masking based analysis and iteratively rewrites them to propose improved variants. Crucially, during evaluation, PMPO selects among variants by minimizing loss in a single forward pass, eliminating output sampling and human or judge based scoring for selection while still using standard generation only to propose rewrites. This unified, loss based strategy supports both supervised and preference based tasks. Across model sizes and datasets, PMPO outperforms prior prompt optimizers: it achieves the highest average accuracy on BBH, performs strongly on GSM8K and AQuA RAT, and raises AlpacaEval 2.0 win rates by over 19 points. These results demonstrate PMPO’s effectiveness, efficiency, and broad applicability.

## introduction prompt design has emerged as a critical factor in steering large language models (llms) toward reliable performance across diverse tasks. as fine-tuning becomes increasingly costly or restricted , automatic prompt optimization has become a practical alternative to improve model behavior without modifying parameters (agarwal et @xcite . recent methods tackle this by iteratively refining prompts through model feedback @xcite , human preferences @xcite , or reward-based search @xcite . while effective, these approaches often face three main challenges: (1) high cost due to output generation and evaluation loops, (2) reliance on large models capable of introspection or multi-step reasoning, and (3) lack of generality across task types and model sizes. some methods, such as promptwizard @xcite and promptagent @xcite , rely on model-internal critique and iterative analysis, achieving strong results on complex tasks but requiring significant computation and model cooperation. others like bpo @xcite and pas @xcite improve efficiency via offline learning of prompt-rewriting models, but require substantial labeled data or preference annotations, and their applicability to new tasks or small models is limited. spo @xcite avoids ground truth by using the model's own judgment to compare outputs, but this self-evaluation is less reliable for smaller lms. a central limitation across these approaches is how prompt quality is evaluated. most techniques treat prompt evaluation as a generative task, where outputs are generated and then scored externally, making the process computationally expensive and less reliable. external scoring often lacks consistency or sufficient granularity, especially when relying on models to assess complex outputs. moreover, methods tend to specialize: some focus on supervised tasks with explicit labels, while others target alignment tasks defined by preferences or style. few offer a unified, efficient mechanism applicable to both. in this work, we introduce pmpo (probabilistic metric prompt optimization), a novel prompt optimization framework that directly uses the model's cross-entropy loss as the evaluation signal. pmpo frames prompt optimization as a classic loss minimization problem: given a prompt and input, it computes the likelihood of the desired or preferred output without requiring the model to generate or rank outputs explicitly. this evaluation is efficient, fully automated, and compatible with models of varying scales, making pmpo applicable to both large and small language models in diverse deployment settings. moreover, pmpo supports both accuracy-based tasks (e.g., classification, qa) and preference-based tasks (e.g., summarization, instruction following) under a unified loss-based evaluation strategy: it minimizes cross-entropy loss for tasks with labeled outputs and maximizes the likelihood of preferred responses over less preferred ones when comparative preference information is available. figure 1 presents an overview of the pmpo refinement loop and summarizes its advantages over existing approaches. compared to prior methods, pmpo offers the following key advantages: • loss-based evaluation. unlike prior methods, pmpo's evaluation of candidate prompts requires no output sampling or human evaluation, relying only on a single forward pass to compute log-likelihoods, which makes its evaluation process highly efficient. ## pmpo framework we introduce pmpo, a unified framework designed to iteratively refine prompts using fine-grained cross-entropy evaluation and adaptive rewriting strategies (see figure 2 ). pmpo systematically analyzes prompts by applying a model-adaptive masking technique to quantify token-level impacts (step 1, detailed in section3.2), and selectively rewrites the lowest-performing segments using language model-generated variants (step 2, described in sections3.3 and3.4). through an iterative cycle of prompt generation, evaluation, and refinement (step 3, corresponding to the overall loop in sec-tion3.1), pmpo automatically enhances prompt effectiveness without human intervention, adapting flexibly across different model architectures and tasks. problem definition given a task t , our framework initializes with a base instruction p and leverages a problem description paired with a dataset @xmath0, y i , r i ) n @xmath1, where each example consists of an input x i , an output y i , and an associated la-bel or preference score r i indicating the quality or desirability of y i for x i . the language model m generates outputs with probabilities p m (y | x, p) conditioned on input x and instruction p. our objective is to derive an optimal prompt p * that maximizes the expected weighted log-probability, that is, e (x,y,r)∼d [r • log p m (y | x, p)], where r can represent either a binary label or a scalar preference score. concretely, in accuracy-based tasks (e.g., qa/classification) we set r i ≡ 1, whereas in preference-based tasks we form pairs and assign @xmath2. this unified formulation enables the framework to enhance the model's target-aligned generation capability by amplifying preferences over desired outputs. ## conclusion we present pmpo, a unified and efficient framework for prompt optimization that relies on lossbased evaluation and iterative rewriting. instead of relying on output generation or human feedback, pmpo uses token-level likelihoods to identify and refine underperforming prompt segments, enabling scalable and efficient optimization. experimental results on reasoning, mathematical, and instructionfollowing benchmarks show that pmpo consistently outperforms existing methods in both accuracy and efficiency. its lightweight design, compatibility with smaller models, and minimal need for manual supervision make it well-suited for both academic research and practical deployment.