Quality Estimation and Post-Editing Using LLM s For I ndic Languages: How Good Is It? (2025)

Abstract: Recently, there have been increasing efforts on Quality Estimation (QE) and Post-Editing (PE) using Large Language Models (LLMs) for Machine Translation (MT). However, the focus has mainly been on high resource languages and the approaches either rely on prompting or combining existing QE models with LLMs, instead of single end-to-end systems. In this paper, we investigate the efficacy of end-to-end QE and PE systems for low-resource languages taking 5 Indian languages as a use-case. We augment existing QE data containing multidimentional quality metric (MQM) error annotations with explanations of errors and PEs with the help of proprietary LLMs (GPT-4), following which we fine-tune Gemma-2-9B, an open-source multilingual LLM to perform QE and PE jointly. While our models attain QE capabilities competitive with or surpassing existing models in both referenceful and referenceless settings, we observe that they still struggle with PE. Further investigation reveals that this occurs because our models lack the ability to accurately identify fine-grained errors in the translation, despite being excellent indicators of overall quality. This opens up opportunities for research in end-to-end QE and PE for low-resource languages.

## introduction the rapid advancements in large language models (llms) @xcite @xcite have significantly impacted machine translation (mt) leading to stateof-the-art translation quality. this quality is usually measured at the corpus level using a variety of quality estimation @xcite metrics among which comet (supervised) and gemba (prompting-based) @xcite are known to be the best. specifically, comet has spurred research into language-family specific versions of comet like in the case of indic langugaes @xcite . closely related is the problem of post-editing where once a poor quality translation has been detected, mistakes in translation need to be suitably fixed @xcite . recently, @xcite have shown that it is possible to take error annotations of comet models and the power of synthetic explanations generated by gpt-4, to develop a system that can post-edit erroneous translations thereby improving translation quality. their main focus was showing that error explanations in human understandable formats lead to improved post-edits by llms. on the other hand, @xcite have leveraged llms purely in prompting mode in multiple stages to first annotate errors, choose the most reliable ones, and then post-edit to improve translation quality. however, existing works have two major limitations: a. they do not focus on a singular endto-end model which does error annotations, error explanations and post-editing in one go. b. they focus on high-resource languages, which makes it difficult to determine the impact on low-resource languages. in this paper we attempt to fill this gap by focusing on english to indian languages (@xmath0: hindi, gujarati, marathi, malayalam and telugu, which are considered low-resource in the world of quality estimation and post editing. given the lowresource setting, we ask a simple question: how good is an all-purpose end-to-end error annotation, explanation and post-editing system for indian languages in a low-resource setting?. this leads to 3 specific research questions (rqs): (rq1): how well do large language models per-you are provided with a source, translation and a translation quality analysis. the translation quality analysis contains a translation with marked error spans with different levels of severity (very low, low, default, high, very high) along with the error type (accuracy (addition, mistranslation, omission, untranslated text), fluency (character encoding, grammar, inconsistency, punctuation, register, spelling), style (awkward), terminology (inappropriate for context, inconsistent use), non-translation, other, source error, transliteration, and no-error). given this information, generate an explanation for each error and a fully correct translation. you have to correct all the errors pointed out in the error span. if the sentence has multiple error spans, write explanations of all the errors together, in a single paragraph. if there is no error pointed out, the corrected translation will be same as translation. form in evaluating machine translation quality for indian languages, considering both reference-based and reference-free scenario? (rq2): do explanations of errors and error span detection by llms lead to demonstrable improvements in post-editing performance for indian languages? (rq3): does joint qe and pe, affect qe? taking motivation from (treviso et al., 2024), we augment the indic mt evaluation dataset @xcite with synthetic explanations and post-edits (see figure 1 ) and fine-tune gemma2 @xcite to obtain a single model to generate error annotations (used for computing mqm scores for qe), error explanations and post-edits. on the positive side, we find that qe significantly surpasses all existing models like cometkiwi, however, unlike previous works, we observe that error annotation and explanation does not often lead to higher translation quality after post-editing. upon further investigation, we find that this mainly occurs because the limited amount of training data leads to models, which are good at evaluating overall translation quality, but are not always reliable at fine-grained quality estimation. specifically, they tend to under-detect certain error categories or sometimes misclassify errors, leading to inconsistencies in post-editing corrections. this shows that we are still far away from using llms for fine-grained error annotation and use it for postediting in low-resource settings. our contributions are: (i) state-of-the-art quality estimation models for 5 indian languages in the @xmath1. (ii) augmented quality estimation dataset with error explanations and gpt4 post-edits. (iii) a reality check that llms are still unreliable for fine-grained quality estimation and post-editing in low-resource settings. ## related work research in the machine translation (mt) evaluation has evolved significantly, driven by the need for more accurate and interpretable metrics. traditional mt evaluation metrics can be broadly classified into reference-based and reference-free approaches. early metrics, such as bleu @xcite and chrf @xcite , primarily relied on lexical overlap between machine translations and human references, often failing to align well with human judgments. more recent neural based metrics like, comet @xcite and bleurt @xcite have shown stronger correlations with humans, but these metrics lack interpretability. these metrics have further improved with the introduction to models like xcomet @xcite and cometkiwi ( in reference-free direction) @xcite . however, xcomet primarily detects error spans and their severity without classifying the specific type of error. we aim to explore whether llms can capture fine-grained translation errors by identifying their types alongside assessing severity, focusing on indian languages. in parallel, the exploration of large language models (llms) for mt evaluation has gained momentum @xcite , with research examining their effectiveness in assessing translation quality. while these approaches have been widely explored for high resource languages, their performance for indian languages, which are notoriously resource poor for quality estimation, remains unexplored. additionally, research suggests that fine-grained error analysis and explanations can improve postediting efficiency @xcite . however, our findings indicate that such ## methodology our approach leverages synthetic explanations and post-edits from llms followed by fine-tuning open-source llms to enhance a large language model's (llm) ability to detect, explain, and correct machine translation errors in both referencebased and reference-free settings. ## experimental setup we now describe specifics of our experimental setup, namely datasets and languages, baselines, model configurations we tested, qe metaevaluation and pe evaluation approaches. ## result in this section, we present the evaluation results of our llm-based approach for mt quality assessment of indian languages, addressing the research questions outlined in section 1. section 5.1 addresses rq1 by evaluating the performance of our models under both reference-based and referencefree settings, comparing them against state-of-theart mt evaluation systems. section 5.2 focuses on rq2, investigating whether error annotations and explanations enhance post-editing performance. additionally, throughout both sections, we explore rq3, analyzing whether joint quality estimation (qe) and post-editing (pe) influence qe performance. by structuring our results around these questions, we provide a comprehensive assessment of llm capabilities for low-resource mt evaluation. ## conclusion our study investigates the role of large language models (llms) in machine translation (mt) evaluation for indian languages, addressing key challenges in fine-grained quality estimation (qe) and post-editing (pe). we leveraged synthetic error explanations and post-edits from gpt-4 and finetuned the gemma-2-9b model in a variety of settings for reference-based qe and reference-free qe and pe. in reference-based settings we got comparable if not slightly better qe performance against existing strong baselines. on the other hand, in reference-free settings we obtained significantly improved qe performance. however in the case of pe, contrary to previous works in high-resource settings, involving error detection and explanation in the pe framework does not lead to improved post-edited translations. the explanation for this is in the poor fine-grained error detection capabilities of our fine-tuned models due to low-resource settings. this indicates a dire situation but opens avenues for future research on joint qe and pe for low-resource languages.