P rompt O pt M e: Error-Aware Prompt Compression for LLM -based MT Evaluation Metrics (2025)

Abstract: Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model’s outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a 2.37\times reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.

## introduction the rapid advancement of natural language generation (nlg) technologies has led to an increasing reliance on automated systems for producing human-like text across various domains, including machine translation (mt). as these systems become more prevalent, the need for effective and efficient evaluation metrics to assess generated content quality has also grown. evaluating nlg systems is a fundamental figure 1 : the two-stage model training approach used in promptoptme. at the first stage, the model is finetuned in a supervised way to adapt it for the compression task and prompt format. at the second stage, we utilize preference data obtained through evaluation of compressed prompts to train the model to select the best compression for each example. subsequently, trained evaluation metrics such as comet @xcite , bleurt @xcite , and xcomet @xcite emerged. these models are trained on human-annotated datasets to predict quality scores, allowing them to better align with human judgments. despite their improved quality, these metrics require substantial amounts of labeled data for training and may not generalize well across different tasks or domains. with the advent of large language models (llms) like @xcite , researchers have started to explore their potential for nlg evaluation through prompting instead of traditional training. metrics such as gemba-da @xcite , gemba-mqm @xcite , automqm @xcite , and g-eval @xcite utilize llms by providing instructions and few-shot examples within prompts. this approach offers several key benefits: a) elimination of task-specific training, as llms can perform evaluations based on prompts without the need for labeled datasets; b) high-quality results, with llm-based metrics demonstrating superior alignment with human judgments and more effectively capturing linguistic nuances and contextual information; c) flexibility and generalization, since these models can be easily adapted to different tasks and domains by modifying prompts, offering greater flexibility compared to traditional trained metrics. despite these advantages, using llms like gpt-4 for evaluation is computationally expensive due to the extensive token usage required by elaborate prompts. for instance, the original gemba-mqm prompt typically requires between 1100 and 1200 tokens per example, accounting for exhaustive instructions and three few-shot examples. if we aim to evaluate this metric on 60k examples from the wmt22 metrics challenge test set @xcite , the total token usage would be on the order of: total tokens ≈ 60k × 1200 ≈ 72m tokens. given that gpt-4 pricing is $10 per million tokens used, the estimated cost for evaluating the entire dataset would be: these substantial costs present a barrier to practical deployment, especially in budgetconstrained settings. it also makes llm-based metrics like gemba-mqm egregiously expensive for large-scale evaluation scenarios, like online re-ranking of mt-systems outputs or web-scale dataset processing @xcite . to address these challenges, we introduce promptoptme, foot_0 a novel approach to prompt optimization that reduces token usage and computational costs in llm-based evaluation by utilizing a smaller, fine-tuned language model to compress input data without sacrificing necessary information. we summarize our contributions as follows: a) we propose a twostage fine-tuning process for promptoptme, involving supervised fine-tuning and preference optimization @xcite , to refine the model's outputs based on actual metric behavior with compressed inputs. b) we apply promptoptme to mt, achieving a 2.32× reduction in token usage for mt evaluation, without any loss in evaluation quality. our work enhances the accessibility of advanced llm-based evaluation metrics by making them more cost-effective and efficient for broader use in real-world nlg applications, thereby promoting diversity and inclusion in nlp research and application by enabling participation from under-resourced communities (belouadi and eger, 2023). ## related work the field of prompt optimization has seen substantial activity in recent years. this section covers works related to prompt optimization, efficient evaluation, and explainable metrics, highlighting how our approach builds upon and differs from existing research. we begin by examining the landscape of explainable evaluation metrics @xcite @xcite . @xcite propose a taxonomy for such metrics, within which the gemba-mqm metric falls into the category of fine-grained error metrics. naturally, these metrics critically depends on preservation of error-spans within source text and translation. unlike xcomet @xcite , gemba-mqm's sentence-level scores are derived solely from extracted errors and their severities, adhering to the mqm guidelines @xcite . due to this fact, we designed our method to fine-tune the model to preserve the error-spans in the source and translation text by utilizing the wmt mqmannotated dataset. further, promptoptme can also be seen as explaining prompt-based evaluation metrics because it yields additional insights into which prompt parts are relevant for a metric to perform with high-quality. in the realm of prompt compression and optimization, several notable approaches have been developed. llmlingua @xcite introduces a coarse-to-fine prompt compression method, employing a budget controller and an iterative token-level compression algorithm. this approach utilizes a smaller language model to compute token importance, substantially reducing token usage and inference latency across various tasks. building upon this foundation, llmlingua-2 @xcite proposes a data distillation procedure to derive knowledge from llms for efficient and faithful task-agnostic prompt compression. by formulating prompt compression as a token classification problem and leveraging a transformer encoder, llmlingua-2 further improves compression efficiency and generalizability. unlike both of those approaches, we specifically target the task of llm-based evaluation metrics. we incorporate preference optimization based on actual metric behavior, tailoring the prompt compression to preserve essential evaluation information. black-box prompt optimization (bpo) @xcite applies prompt optimization for black-box llm alignment, using a prompt preference optimizer trained on human preference data. this model-agnostic approach enhances the alignment of llm outputs with human intents without requiring access to model parameters or additional training. our method shares similarities in leveraging preference learning, but we extract these preferences by exploiting measurable quality differences during prompt execution with optimized outputs. prewrite @xcite proposes an automated prompt engineering approach using reinforcement learning to rewrite prompts for improved prediction quality. while both prewrite and our method optimize prompts without modifying the underlying language model, our approach distinguishes itself by simultaneously striving to reduce prompt size and improve quality. in the domain of efficient mt evaluation metrics, several approaches have been proposed to address the computational challenges posed by large trained metrics. xcomet-lite @xcite , cometinho @xcite , and frugalscore @xcite all employ techniques such as pruning, quantization, and distillation to create smaller, more efficient models. while these methods achieve impressive compression ratios, resulting in models with under 500m parameters, they often experience a drop in correlation with human judgment. our approach aims to mitigate this quality loss by continuing to rely on large backbone llms while improving efficiency through targeted prompt optimization. @xcite explore an alternative approach to creating an efficient version of bertscore and moverscore by replacing the underlying encoder model with smaller alternatives, such as pruned or distilled models. we touch this subject too, by evaluating our prompt compression model not only on large backbone llm gpt-4o, but also on smaller models: gpt-4o mini and llama 3.2. as we demonstrate in section 5, our approach proves effective across different model sizes, improving evaluation efficiency even on smaller llms. ## method in this section, we detail our two-stage approach for prompt optimization aimed at reducing token usage and computational costs in llm-based evaluation metrics. we first look into the optimization of prompt inputs (source texts and machine translations) and later detail our approach to compressing the rest of the prompt. the first stage involves supervised fine-tuning of a language model to learn the specifics of the prompt compression task. the second stage employs preference optimization using the odds-ratio preference optimization (orpo) algorithm @xcite to refine the model's outputs based on the observed behavior of the metric. we chose this algorithm due to its specific focus on reducing the likelihood of generating the rejected responses, which are, in our case, low-quality compressed source and translation texts. the overall approach is visualized in figure 1 . ## experiments in this section, we evaluate the effectiveness of our proposed prompt optimization approach in reducing token usage while maintaining evaluation quality in llm-based evaluation metrics. we conduct experiments on mt evaluation and assess the compression efficiency and the impact on evaluation quality. as a base model for fine-tuning, we utilize the llama-3.2 @xcite model in two variants -1b and 3b parameters -chosen for its extensive vocabulary and multilingual pre-training beneficial for handling diverse language pairs. stage one: supervised fine-tuning for the first stage, we use the wmt metrics shared task datasets with mqm annotations from the years 2020 to 2022 @xcite . we exclude the 'news' portion of wmt22 dataset for validation purposes. the remaining training data consists of approximately 145k examples across three language pairs: english-russian, english-german, and chinese-english. following the approach described in section 3, we construct prompts with system instructions for the model. the prompt follows this template: you are a helpful ai assistant that intelligently compresses and summarizes the given machine translation outputs for further evaluation. compress the following mt input and output: source:"'{src-text}"' mt:"'{mt-text}"' the expected model completion is: quality-relevant parts of source: [error-span-1]; ... ; @xcite or none quality-relevant parts of translation: stage two: preference optimization for the second stage, we construct a preference dataset by selecting a subset of 20k examples from the training data. for each example, we generate compressed versions at each compression rate from our predefined set, resulting in eight different compressions per example. we incorporate these compressed examples into the original gemba-mqm prompt and evaluate them using the gpt-4o model to obtain evaluation scores. based on these scores, we select the chosen and rejected compressions following the procedure described in section 3. using this preference dataset, we fine-tune the model from stage one employing orpo. we train for three epochs with a batch size of 64, a learning rate of 1 × 10 -5 , and cosine warmup over 6% of training steps. we set the orpo λ parameter to 0.1 following the original paper authors. evaluation procedure to evaluate the effectiveness of our model, we apply it to compress the examples in the test set. this compression is performed on both the few-shot examples and the target example within the gemba-mqm prompts. we then conduct mt evaluation using these compressed prompts with proprietary llms gpt-4o and gpt-4o-mini and the openly-available llama-3.2-vision model with 90b parameters. in our experiments, we utilize both the original and simplified prompts (as described in section 3.3) and generate outputs in both plain text format and in json. we assess the quality of the evaluations by computing pairwise accuracy at the system level, as suggested by @xcite , comparing our results with human judgments. additionally, we calculate segment-level pearson r and kendallτ correlation for each language pair to measure the agreement between our evaluations and human assessments. in addition to the validation set, we use wmt23 metrics challenge @xcite dataset to perform meta-evaluation of our approaches and compare them with existing metrics. to quantify the efficiency gains, we measure the cumulative token usage during the evaluations. this provides insights into the computational cost savings achieved through our prompt optimization approach. to compare the effectiveness of our approach, we also include two baseline results obtained using the llmlingua-2 @xcite prompt compression method. in those cases, we have applied microsoft/llmlingua-2-xlm-robertalarge-meetingbank to compress source texts and machine translations similarly as we do with promptoptme. we fix and test two compression rates: 30% and 50%. ## results in this section, we present the results of our experiments evaluating the effectiveness of our proposed prompt optimization approach. by applying our prompt optimization model, promptoptme-3b, with gpt-4o lite prompt template, we achieve the highest reduction rate of 2.37×, reducing token usage from 19m to 8.3m tokens. surprisingly, promptoptme-3b attains the best segment-level kendall τ correlations across all language pairs, with 0.4455 for en-ru, 0.4065 for en-de, and 0.3738 for zh-en, fully recovering reduced scores in the baseline, while also retaining the same level of system-level pairwise accuracy. this trend continues with other backbone llms. for instance, using promptoptme-3b with gpt-4o mini lite prompt, we, again, achieve the same reduction rate of 2.37× and observe an improvement in pairwise accuracy to 0.7842, which is a 2.77% increase over the gpt-4o mini ref baseline. however, the segment-level correlations exhibit mixed results: while there is a substantial decrease for en-ru (14.67% lower than the baseline), there is a positive improvement for en-de (an increase of 2.30%) and zh-en (a 3.57% increase). similarly, for the llama3.2-90b model, applying promptoptme-3b results in a reduction rate of 2.27×. the pairwise accuracy remains the same as the baseline (0.7526), but the segment-level kendall τ correlations show modest improvements for en-ru (a 2.61% increase) and en-de (9.28% higher), while zh-en experiences a decrease of 12.68%. thus, overall, we see substantial efficiency gains and simultaneously often an increase in quality of the resulting metrics; however, there are cases when metric quality reduces, sometimes considerably. in comparison with the baseline approaches with the llmlingua-2, we see dramatically decreased quality in both compression settings, as compared to our approach and baseline metrics. for the system-level pairwise accuracy, we observe a 31%-40% decrease from the original uncompressed metric. on a segment level, the quality decrease is even more catastrophic. for two out of three language pairs, the correlation plunges to near zero. for zh-en, it drops to 0.19-0.12. this indicates that llmlingua-2 substantially damages the prompt to the point of complete non-usability. in ## discussion our proposed prompt optimization model, promptoptme, demonstrates substantial improvements in computational efficiency for machine translation evaluation. by integrating promptoptme with a simplified prompting approach, we achieve substantial reductions in token usage-up to 2.37× less than the baseline-while maintaining or even improving evaluation quality across various metrics. these findings suggest that it is possible to compress inputs substantially while maintaining evaluation quality. we speculate that outputs of few-shot examples, which are not compressed in any way in our experiments, play a crucial role in establishing model quality in mt evaluation, as they themselves cover all error severity levels as well as error categories from mqm typology. preserving entire source and translation texts, in turn, appears to be redundant. we are able to retain quality while preserving only part of the text, which includes error-spans. the superior quality of promptoptme-3b, compared to the smaller promptoptme-1b model, indicates that a larger prompt optimization model is more effective at compressing input data efficiently. specifically, promptoptme-3b refers to our prompt optimization model based on llama 3.2 with 3 billion parameters, achieves 10% (2.37 vs. 2.15) higher compression rate on average, along with 2%-3% higher pairwise accuracy, compared to promptoptme-1b. it demonstrates the importance of model capacity in the compression process. further experiments with larger llms could potentially help get a better understanding of scaling laws for that particular task. when applying promptoptme-3b to different backbone llms, such as gpt-4o mini and llama3.2-90b, we observe varying results. for gpt-4o mini, we achieve an improved pairwise accuracy compared to the baseline, with a reduction rate of 2.37×. in contrast, when applying promptoptme-3b to llama3.2-90b, the pairwise accuracy remains consistent with the baseline, and we observe modest improvements in segment-level kendall τ correlations for some language pairs. in both cases, we notice a slight degradation in segment-level correlations in one of the language pairs, however. nonetheless, we can conclude that our prompt optimization model achieves generalization across different backbone llms, despite being trained only on preferences obtained through gpt-4o. additionally, compared to baseline approaches using llmlingua-2, our method clearly outperforms in both system-level and segmentlevel evaluations. the substantial quality degradation observed with llmlingua-2, particularly at the segment level, indicates that it may not be suitable for effective prompt compression in mt evaluation tasks. this result is in agreement with our hypothesis that preserving quality-relevant spans in text is essential for maintaining evaluation quality. llmlingua-2, as a non-task-specific prompt compression method, is not trained to take that into account. evaluation on the wmt23 metrics challenge further supports the robustness of our prompt optimization model. the preserved pairwise accuracy and high pearson correlation indicates that our approach generalizes across different datasets and language pairs. future work may investigate adaptive compression strategies to better handle low-resource or previously unseen language pairs. ## conclusion in this paper, we introduced promptoptme, a prompt optimization approach designed to reduce token usage and computational costs in large language model-based evaluation metrics. by leveraging a smaller, fine-tuned language model to compress input data, we achieved substantial reductions in token usage without compromising evaluation quality. specifically, promptoptme reduced token usage by up to 2.37× while model + prompt maintaining or improving segment-level kendall τ correlations across multiple language pairs as well as system-level pairwise accuracy. our approach enhances the practicality of llm-based evaluation metrics, making them more cost-effective and accessible for large-scale natural language generation (nlg) applications. by addressing the computational expense associated with extensive prompt token usage, promptoptme enables more efficient evaluations. this reduction in computational cost makes state-of-the-art mt evaluation more accessible to under-resourced researchers and students, who may have limited access to computational resources or funding for extensive llm usage. by lowering the barriers to highquality evaluation, promptoptme democratizes the ability to conduct advanced research and development in machine translation. our approach is especially useful in this case, as it does not require a compromise in metric quality, offering no quality drop while decreasing the token usage. moreover, we are able to effectively compress the inputs for smaller llms as well, such as gpt-4o mini and llama 3.2 90b, which makes sota evaluation metrics even more accessible. while our experiments focused on machine translation, where error highlights and annotations are common, we acknowledge that extending this approach to other nlg tasks may present challenges, particularly in scenarios where such detailed error annotations are not available. future work could explore adapting promptoptme to other nlg evaluation tasks, investigating how input compression affects evaluation quality in contexts without explicit error spans, and determining the generalizability of our method across various domains. it is also important to note, that the optimized prompts generated by promptoptme are not guaranteed to strictly adhere to the mqm error typology. this divergence from the standard evaluation framework may impact the consistency and interpretability of the evaluation results, especially when comparing outputs across different systems or studies. future work should focus on enhancing the alignment of compressed prompts with established evaluation standards to improve the reliability and comparability of the assessments. we plan to release the code and models to the public at https://github.com/nl2g/ promptoptme to facilitate further research and application in this area. we believe that promptoptme offers a promising direction for efficient high-quality llm-based evaluations. second, we evaluated our method using only one openly available language model in addition to gpt-4o, specifically the llama-3.2 model. while the results are promising, testing our approach on a wider variety of models, including more open-source and proprietary llms with diverse architectures and sizes, would help establish the generalizability and robustness of promptoptme. future work should include experiments with additional models to better understand the applicability of our prompt optimization technique. additionally, our approach relies on a twostage fine-tuning process involving supervised finetuning and preference optimization. this process requires access to sufficient training data and computational resources for fine-tuning the smaller language model used for prompt compression. in scenarios where such resources are limited or unavailable, the practicality of our method may be constrained. exploring alternative approaches that require less extensive fine-tuning or that leverage zero-shot or few-shot learning could mitigate this limitation. in addition to one-time training cost, practitioners using promptoptme should also consider inference costs for the compression model itself. further work should take these costs into account for more fair comparison with the baseline. furthermore, while our prompt optimization aims to preserve essential evaluation information, there is a possibility that some fine-grained details, such as specific mqm error categories, may not be fully captured in the compressed prompts. this could potentially affect the precision in identifying certain types of translation errors, leading to less detailed evaluations. ensuring that critical information is retained during compression without increasing token usage remains a challenge that warrants further investigation.