Don’t Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation (2024)

Abstract: Neural machine translation systems estimate probabilities of target sentences given source sentences, yet these estimates may not align with human preferences. This work introduces QE-fusion, a method that synthesizes translations using a quality estimation metric (QE), which correlates better with human judgments. QE-fusion leverages a pool of candidates sampled from a model, combining spans from different candidates using a QE metric such as CometKiwi. We compare QE-fusion against beam search and recent reranking techniques, such as Minimum Bayes Risk decoding or QE-reranking. Our method consistently improves translation quality in terms of COMET and BLEURT scores when applied to large language models (LLMs) used for translation (PolyLM, XGLM, Llama2, Mistral, ALMA, and Tower) and to multilingual translation models (NLLB), over five language pairs. Notably, QE-fusion exhibits larger improvements for LLMs due to their ability to generate diverse outputs. We demonstrate that our approach generates novel translations in over half of the cases and consistently outperforms other methods across varying numbers of candidates (5–200). Furthermore, we empirically establish that QE-fusion scales linearly with the number of candidates in the pool.

## introduction neural machine translation (nmt) models are probability estimators of translations given source sentences. therefore, errors in nmt often arise due to the misalignment between the model's probabilities and human preferences @xcite @xcite . in contrast, mt evaluation metrics have shown increasing correlation with human judgements @xcite @xcite , making them valuable tools for selecting candidates generated by nmt models. reference-based evaluation metrics such as comet @xcite and bleurt @xcite have been successfully employed for minimum bayes risk (mbr) decoding @xcite . in mbr, these metrics serve as utility functions to select the candidate with the highest similarity to all other candidates @xcite . additionally, quality estimation (qe) metrics such as comet-qe @xcite , which do not need reference translations, have been used to rerank a pool of candidates based on their estimated quality @xcite . such reranking approaches improve translation quality over standard beam search, particularly when measured with neural-based mt evaluation metrics. however, reranking methods are challenged in situations where candidate translations exhibit complementary errors, with no candidate clearly improving over all others. figure 1 illustrates this issue. the first candidate, fire in french chemical plant, uses in instead of the more idiomatic at. the third candidate makes a better choice in this case but renders the verb as cleared instead of extinguished, which is incorrect here. making the most of this pool of candidates is not possible without combining fragments from several ones. to address this limitation, we propose qefusion, an approach that leverages the potential complementarity of model-generated candidates to synthesize an improved translation, guided by a qe metric. qe-fusion starts by identifying divergent spans, i.e. spans that exhibit variations within the pool of candidates. then, it traverses these divergent spans, selecting at each step the one that contributes to a higher score according to the qe metric. the chosen spans are integrated into the synthesized translation, which is thus a fusion of multiple candidates. we demonstrate that qe-fusion is effective with pools of candidates obtained from high-performing the method first generates multiple hypotheses by sampling translations from the model. then, it computes and sorts the spans that diverge among the candidates. finally, a qe metric is used to select a span from each group and these spans are merged to form a new, refined translation. ## related work we review in this section several approaches that aim to incorporate additional knowledge either during the decoding process or after it, in order to mitigate the misalignment between mt models and human preferences. we start with a brief reminder of mt evaluation metrics, which play a crucial role as a selection criterion. ## conclusion in this paper, we introduced qe-fusion, a novel approach that combines fragments from diverse candidates to synthesize improved translations guided by quality estimation metrics. our experiments across five language pairs involving both llms and nmt models demonstrate that qe-fusion consistently outperforms beam search, mbr decoding, and qe-reranking. qe-fusion is particularly beneficial to llms, capitalizing on their diverse outputs. notably, qe-fusion maintains its superior performance even when the pool of candidates grows, while its runtime scales linearly with the number of candidates, highlighting its scalability. by integrating external knowledge from quality estimation metrics, qe-fusion not only enhances translation quality but also tackles specific issues like hallucinations. this offers a promising avenue for addressing other phenomena such as contextspecific errors @xcite . beyond translation, the versatility of qe-fusion extends to any text generation task where metrics or reward models @xcite are available.