Quality-Aware Decoding: Unifying Quality Estimation and Decoding (2025)

Abstract: Quality Estimation (QE) models for Neural Machine Translation (NMT) predict the quality of the hypothesis without having access to the reference. An emerging research direction in NMT involves the use of QE models, which have demonstrated high correlations with human judgment and can enhance translations through Quality-Aware Decoding. Although several approaches have been proposed based on sampling multiple candidate translations and picking the best candidate, none have integrated these models directly into the decoding process. In this paper, we address this by proposing a novel token-level QE model capable of reliably scoring partial translations. We build a uni-directional QE model for this, as decoder models are inherently trained and efficient on partial sequences. We then present a decoding strategy that integrates the QE model for Quality-Aware decoding and demonstrate that the translation quality improves when compared to the N-best list re-ranking with state-of-the-art QE models (up to 1.39 XCOMET-XXL). Finally, we show that our approach provides significant benefits in document translation tasks, where the quality of N-best lists is typically suboptimal. Code can be found at https://github.com/SAP-samples/quality-aware-decoding-translation.

## introduction large language models (llms) have significantly impacted various natural language processing (nlp) tasks @xcite @xcite , including neural machine translation (nmt). the field of nmt is transitioning from using dedicated encoder-decoder transformers @xcite to leveraging decoder-only llm-based translation models @xcite . this shift is driven by llms' ability to retain knowledge, handle large contexts, and follow instructions, learned during extensive pre-training @xcite . as a result, llm-based mt models have achieved state-of-the-art translation quality @xcite . in parallel, quality estimation (qe) has become a well-researched subfield within nmt. qe models are trained to predict the quality of a translation without requiring access to the reference @xcite . interestingly, qe models can achieve performance in assessing translation quality that is comparable to mt evaluation models, which do have access to the reference @xcite . this led to the question: "can we integrate qe into the current translation process to improve quality?" incorporating qe into nmt offers several benefits. first, having a expert qe model guiding the decoding can further improve the quality. second, by adapting the qe model with feedback from human annotators, we can generate future translations guided with the newly obtained feedback. several approaches have been explored to integrate qe into the translation process. these include re-ranking the n-best list @xcite , applying minimum bayes risk (mbr) decoding on a quality-filtered n-best list @xcite , and training additional models for post-editing based on qe-predicted errors @xcite . however, all these methods operate on fully generated sequences before the qe model can exert influence. integrating qe earlier in the decoding process, referred in this paper as quality-aware decoding, could enhance translation quality and reduce reliance on the n-best list. this is especially relevant when dealing with long inputs as figure 1 : example from wmt'23 @xmath0: 10: the paragraph begins with 'department of homeland security,' which should be translated as 'ministerium für innere sicherheit.' however, the top 25 beams do not contain the correct translation and begin with an error, making n-best list re-ranking insufficient. although the top-5 tokens at the decoding contain the correct forms 'inn' or 'inner,' the probabilities split among them giving highest mass to the incorrect token 'inn.' quality-aware decoding can prevent errors with earlier integration. good translations during decoding are likely to be pruned and may need sampling larger number of finished hypothesis. we illustrate this in figure 1 . to achieve this, a qe model capable of predicting the quality of partial translations is required. however, current leading qe models face challenges in this area, as they are typically not trained to predict scores for incomplete hypotheses. therefore, developing qe models that can handle partial translations is essential for implementing quality-aware decoding during the translation process. in this work, we propose adapting llm-based nmt models to perform qe on partial translations and incorporating this model into the decoding. we create a token-level synthetic qe dataset using wmt multidimensional quality metrics (mqm) data @xcite . we then adapt a uni-directional llm-based mt model to predict whether a token is good or bad. training qe models on these token-level tasks alleviates the data challenge and allows us to exploit the mqm data while simultaneously making the task easier for the model compared to predicting a score directly. furthermore, integrating the qe model into nmt during decoding is not trivial, as we need to combine the qe estimates during decoding. therefore, we modify the decoding strategy from @xcite to incorporate token-level predictions efficiently with the adapted qe model to provide real-time feedback during the decoding process. we summarize our contributions below. ## quality-aware decoding the primary objective of this paper is to achieve quality-aware decoding in nmt. to accomplish this, it is essential to predict the quality of partial translations and integrate this information during the decoding process. our approach proposes using one nmt model for generating translations and another adapted nmt model to predict the quality of the candidate translations produced by the first model. first, we explain why relying solely on the nmt model to predict the quality of a hypothesis is insufficient and why an additional model is necessary. next, we outline the adaptation of the nmt model for qe on partial translations, detailing the creation of a token-level qe dataset, the modifications made to the nmt model for this task, and the process of estimating the sentence-level quality score. finally, we describe the algorithm used to incorporate the qe score into the decoding process. ## results we conduct a series of experiments to validate the effectiveness of quality-aware decoding and identify the scenarios where it provides the most benefit. first, we evaluate whether our token-level qe model can better estimate sequence quality compared to the log probabilities of the nmt model. next, we assess the impact of quality-aware decoding by comparing it with other approaches to determine if it improves translation quality. we also perform an ablation study to examine whether training the qe model on errors from the same nmt model enhances its performance. then, we explore the impact of source sentence length to highlight the limitations of n-best list re-ranking. finally, we compare our proposed approach with existing quality-aware decoding strategies and also their inference time to highlight the latency/quality trade-off. ## conclusion we demonstrated the value of quality-aware decoding in improving translation quality without relying on post-generation methods. using mqm data, we built a uni-directional token-level qe model and integrated it into the decoding process. our experiments show measurable quality gains, achieved without adding new training data to the nmt model, highlighting that improvements come only from the decoding approach.