Improved Pseudo Data for Machine Translation Quality Estimation with Constrained Beam Search (2023)

Abstract: Machine translation (MT) quality estimation (QE) is a crucial task to estimate the quality of MT outputs when reference translations are unavailable. Many studies focus on generating pseudo data using large parallel corpus and achieve remarkable success in the supervised setting. However, pseudo data solutions are less satisfying in unsupervised scenarios because the pseudo labels are inaccurate or the pseudo translations differ from the real ones. To address these problems, we propose to generate pseudo data using the MT model with constrained beam search (CBSQE). CBSQE preserves the reference parts with high MT probabilities as correct translations, while the rest parts as the wrong ones for MT generation. Therefore, CBSQE can reduce the false negative labels caused by synonyms. Overall, beam search will prefer a more real hypothesis with a higher MT generation likelihood. Extensive experiments demonstrate that CBSQE outperforms strong baselines in both supervised and unsupervised settings. Analyses further show the superiority of CBSQE. The code is available at https://github.com/NJUNLP/njuqe.

## introduction with the rapid development of machine translation (mt), evaluating the quality of mt outputs becomes increasingly essential. common mt metrics, such as bleu @xcite or ter @xcite , rely on reference translations which are unavailable in most applications of mt systems. quality estimation (qe) is the task of automatically assessing the quality of machine translations at run-time without relying on references @xcite . qe plays an important role in real-world scenarios. for example, qe improves post-editing (pe) workflows by selecting high-quality mt outputs and indicating incorrect as shown in therefore, many studies turn to generate pseudo qe data (pseudo mt outputs and pseudo qe labels) using the large parallel corpus. furthermore, these pseudo data are used to pre-train a supervised model or train an unsupervised model. di-rectqe @xcite uses a masked language model conditioned on source sentence as the translation language model (tlm) to generate pseudo translations. given a parallel pair, @xcite randomly mask reference tokens and replace them with ones sampled from the tlm generation distribution. the replaced tokens are annotated as errors, and the sentence score is calculated as the ratio of replacement. bsqe @xcite directly generates pseudo translations using a neural machine translation (nmt) with the beam search algorithm (bs) and obtains labels by matching these translations with corresponding references using ter tools. although pseudo data solutions achieve remarkable performance in the supervised setting @xcite , they are less satisfying in unsupervised scenarios @xcite due to generation noise. e.g., directqe yields repetition or incomplete translation which is different from the real ones due to its non-autoregressive generation @xcite . furthermore, its negative sampling strategy sacrifices the quality of pseudo translation for accurate pseudo labels. on the other hand, bsqe suffers from inaccurate pseudo labels. as shown at the top of figure 1 , the default beam search yields the synonyms of the reference sentence. however, it is difficult for existing referencebased evaluation methods to match these synonyms correctly, especially when the sentence structure changes too much. to improve the quality of pseudo data, we propose to generate pseudo translations using an mt model with constrained beam search (cbsqe). cb-sqe corrupts translations with reference parts by adjusting the generation probability. cbsqe improves the probability of references when the probability of any reference token is above a threshold so that the less fluent hypothesis will not be mistakenly rewarded. besides, we design an adjustment function that assigns larger improvements to closer reference tokens. this allows us to preserve the main structure of references thereby obtaining accurate labels using ter tools. we show an example at the bottom of figure 1 . our main contributions can be summarized as follows: ## method we first introduce the constrained beam search algorithm, then the training and inference procedures of the qe model. ## conclusion we present a constrained beam search algorithm to generate improved pseudo qe data. the proposed cbsqe preserves the main structure of references by increasing the generation probabilities of reference tokens according to their original nmt probabilities and positions. experiments show that we achieve remarkable performance in both supervised and unsupervised settings. the analysis further shows that cbsqe is efficient, and each part of cbsqe contributes to the improvement.