GEMBA V2: Ten Judgments Are Better Than One (2025)

Abstract: We introduce GEMBA-MQM V2, an MQM-inspired, reference-free LLM evaluation metric for the WMT25 Metrics Shared Task (Subtask 1). Building on GEMBA/GEMBA-MQM, we prompt GPT-4.1-mini to produce structured MQM error annotations per segment. We map annotations to scores with 25/5/1 severity weights (minor punctuation = 0.1). To reduce stochastic variance, each segment is scored ten times and aggregated with a reciprocal-rank weighted average (RRWA) after removing outliers beyond 2\sigma . On the WMT24 MQM test sets, GEMBA-MQM V2 ranks first by average correlation, with strong results across languages and evaluation levels; WMT23 results show comparable performance.

## introduction automatic evaluation is essential for assessing machine translation quality at scale. recent work shows that large language models (llms) can act as effective evaluators when guided by mqm-style prompts @xcite . we revisit this approach for wmt25 and propose gemba-mqm v2 -a more robust extension of the original method. using gpt-4.1-mini and full source-document context, we obtain strong segment-level and competitive system-level correlations on wmt24 and wmt23, while controlling judgment variability via multi-run aggregation. structured json inputs/outputs enable reliable parsing and a clean separation between prompt and payload. ## data and evaluation protocol we use mqm human-annotated test sets from wmt23 and wmt24 (overviews: @xcite ; metrics tasks: @xcite ) as distributed in mt-metrics-eval (google research, 2024), following the mqm standard @xcite . for wmt24 we evaluate english-german (en-de), english-spanish (enes), and japanese-chinese (ja-zh); for wmt23 we use english-german (en-de), hebrew-english (he-en), and chinese-english (zh-en). we follow the official task scripts to compute systemand segment-level correlations and report the prescribed measures for each year and language pair. as in the original gemba approach, we use gpt-4.1-mini without further training. ## prompts gemba-mqm v2 prompts gpt-4.1-mini with: (a) an mqm system instruction defining severity (critical/major/minor) and error types, while providing full source-document context and (b) lineby-line json inputs carrying source, target, and language tags. the model returns a json object with lists of errors by severity and type, which we score as mqm "badness" with weights 25/5/1 (minor @xmath0.1), then negate so higher is better for mt-metrics-eval. the mqm protocol for human annotators is based on error span annotation. span marking is difficult for generative llms, so following @xcite we elicit short error descriptions rather than spans. scoring depends only on error severity and error type. figure 1 shows the system prompt and the full english context excerpt used in the example. we then process each text segment individually (split on newlines), as shown in figure 2 , which presents a concrete input/output json pair from a wmt24 document and one of its system outputs. currently, we do not maintain the history of previous segments, only the system prompt is visible for each segment prompt. relative to prior gemba-mqm, we switch to gpt-4.1-mini, enforce json outputs, set temperature to 0.4, and judge line-by-line with full document context present in the prompt. you are an annotator for the quality of machine translation. your task is to identify errors and assess the quality of the translation. to accomplish this, you will receive a pair of paragraphs from this context as a json structure. for each input, reply with an extended json object that contains the following information: focus on errors in the translation, not in the source. each error is classified as one of three categories: "critical", "major", and "minor". critical errors inhibit comprehension of the text. major errors disrupt the flow, but what the text is trying to say is still understandable. minor errors are technically errors, but do not disrupt the flow or hinder comprehension. for every of the main three categories, additionally identify error types in the translation and sub-classify them. the types of errors are: "accuracy" ("addition", "mistranslation", "omission", "untranslated text"), "fluency" ("character encoding", "grammar", "inconsistency", "punctuation", "register", "spelling"), "style" ("awkward"), "terminology" ("inappropriate for context", "inconsistent use"), "non-translation", or "other". for every error type, do also supply a short description ("desc") of the error type. if there are no errors of a specific main category (critical, major or minor), it is ok to return an empty list for that category. it also ok to not return any errors for any category if everything is fine. here is an example of a json input (potentially other languages): { "source_language": "english", "source": "i do apologise about this, we must gain permission from the account holder to discuss an order with another person, i apologise if this was done previously, however, i would not be able to discuss this with yourself without the account holders permission.", "target_language": "german", "target": "ich entschuldige mich dafür, wir müssen die erlaubnis einholen, um eine bestellung mit einer anderen person zu besprechen. ich entschuldige mich, falls dies zuvor geschehen wäre, aber ohne die erlaubnis des kontoinhabers wäre ich nicht in der lage, dies mit dir involvement." } and here is a corresponding json output with example error annotations (potentially other languages) { "source_language": "english", "source": "i do apologise about this, we must gain permission from the account holder to discuss an order with another person, i apologise if this was done previously, however, i would not be able to discuss this with yourself without the account holders permission.", "target_language": "german", "target": "ich entschuldige mich dafür, wir müssen die erlaubnis einholen, um eine bestellung mit einer anderen person zu besprechen. ich entschuldige mich, falls dies zuvor geschehen wäre, aber ohne die erlaubnis des kontoinhabers wäre ich nicht in der lage, dies mit dir involvement.", "errors": { "critical": [], "major": [ {"type": "accuracy/mistranslation", "desc": "'involvement' is untranslated"}, {"type": "accuracy/omission", "desc": "'the account holder' is missing"} ], "minor": [ {"type": "fluency/grammar", "desc": "'wäre' is a bit awkward"}, {"type": "fluency/register", "desc": "'dir' should be 'sie'"} ] } } you should mimic the format from this example. apart from that, you are receiving the full english document as context which will help you analyze the individual json segments provided after this for errors. use this information to analyze the translation pairs in their full context. english input in context: ---siso's depictions of land, water center new gallery exhibition "people swimming in the swimming pool" from 2022 is one vicente siso artwork that will display at tierra del sol gallery beginning jan. 13. (photo courtesy of vicente siso) tierra del sol is pleased to present "vicente siso: memories of the land and water" at the new gallery location in west hollywood. siso has been an artist in the studio arts program since 2012, this marks his debut solo exhibition. siso was born 1962 in madrid and raised between venezuela, trinidad and miami; he moved with his family to southern california in his early 20s. masterfully working across subject matter, siso has generated a prolific series of landscapes, portraits, and still-life works rendered in either acrylic, pastel, pencil or watercolor. drawing from family portraits, his own reference photographs, and recollection, his colorful compositions demonstrate his range of interests and skill across media. siso's tropical landscapes and seascapes reflect the geographies of his past, employing rich patterns and incorporating people to make meaningful connections between culture, memory and the environment. siso titles his artworks in a mix of spanish and english, signifying the celebrated and integral complexities of his life in los angeles county. "vicente siso: memories of the land and water" opens on saturday, jan. 13, with a reception from 6-8 p.m. the exhibition is on view through sunday, march 3. the tierra del sol gallery is located at 7414 santa monica blvd. for information, visit tierradelsolgallery.org. ---next, you will receive the translation pairs as json objects. analyze the translation as discussed above and produce a json object with your analysis in response. do not invent structural elements that are not present in the json examples above. the only allowed keys are "source", "target", "source_language", "target_language", "errors", "critical", "major", "minor", "type", "desc". figure 1 : system prompt with json one-shot example and full document context { "source_language": "english", "source": "siso's depictions of land, water center new gallery exhibition", "target_language": "german", "target": "sisos darstellungen von land, wasserzentrum neuer galerie" } (a) input json object { "source_language": "english", "source": "siso's depictions of land, water center new gallery exhibition", "target_language": "german", "target": "sisos darstellungen von land, wasserzentrum neuer galerie", "errors": { "critical": [ { "type": "accuracy/mistranslation", "desc": "the phrase 'water center new gallery exhibition' is mistranslated as 'wasserzentrum neuer galerie', which incorrectly suggests a 'water center' rather than 'water' being part of the exhibition's focus." }, { "type": "accuracy/omission", "desc": "the word 'exhibition' is omitted; the translation ends with 'galerie' (gallery) without indicating that it is an exhibition. ## aggregation over multiple runs beyond prompt design, we consider aggregation across multiple runs to be the key contribution of gemba-mqm v2 for improving annotation reliability. we score each segment ten times and aggregate the results using a reciprocal-rank weighted average (rrwa, see section a.2): per segment, scores from each run are sorted in descending order, weighted by 1/rank, and outliers beyond 2σ are removed before aggregation. this aggregation strategy is motivated by two observations: we discuss these in more detail below. ## results on wmt24 metrics task data we use the mt-metrics-eval toolkit to compute the correlations reported in single-run ablations group tightly (0.716-0.723), indicating good performance across stochastic runs despite high segment variability reported. appendix tables 3 and 4 provide wmt23 results under 2024 and original settings, respectively. under 2024 rules our gemba-mqm v2 variant would have ranked first on wmt23 data as well. aggregated results improve over each of the individual runs in every category. segment-level performance is especially strong, while system-level performance lags behind. this suggests that the chosen mqm weights and the resulting segment scores may not lend themselves to cross-segment aggregation under equal weighting (simple mean). ## wmt25 submission for our wmt25 subtask 1 submission, we follow the protocol outlined above. for all language pairs and systems we use the same prompts, temperature, and number of stochastic runs as in our wmt24 experiments. the wmt25 unified evaluation task differentiates between language pairs with mqm-style scoring and error span annotation (esa) @xcite . we did not explore the implications of this differentiation in our current submission and simply submitted negated mqm scores for all language pairs. we expect rank-based correlations to carry over under this framework as in prior tasks. at the time this paper was finalized, the shared task organizers had not yet released the final wmt25 metrics task results. as a consequence, we cannot report a definitive leaderboard position for our submission. regretably, this reduces the value of this particular paper. ## conclusion we presented gemba-mqm v2, a reference-free llm evaluation metric/method for (machine) translation that combines json-first prompting, full source-document context, and multi-run aggregation. scoring each segment ten times and aggregating with a reciprocal-rank weighted average (rrwa) improves robustness to stochastic variability and reduces over-annotation effects. on wmt24 mqm test sets, gemba-mqm v2 ranks first by average correlation and is strong across languages and evaluation levels. segmentlevel performance is especially strong, while system-level aggregation remains an open area for improvement.