Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? (2022)

Abstract: Neural machine translation (NMT) is often criticized for failures that happenwithout awareness. The lack of competency awareness makes NMT untrustworthy. This is in sharp contrast to human translators who give feedback or conduct further investigations whenever they are in doubt about predictions. To fill this gap, we propose a novel competency-aware NMT by extending conventional NMT with a self-estimator, offering abilities to translate a source sentence and estimate its competency.The self-estimator encodes the information of the decoding procedure and then examines whether it can reconstruct the original semantics of the source sentence. Experimental results on four translation tasks demonstrate that the proposed method not only carries out translation tasks intact but also delivers outstanding performance on quality estimation.Without depending on any reference or annotated data typically required by state-of-the-art metric and quality estimation methods, our model yields an even higher correlation with human quality judgments than a variety of aforementioned methods, such as BLEURT, COMET, and BERTScore. Quantitative and qualitative analyses show better robustness of competency awareness in our model.

## introduction with the exploitation of large amounts of training data, neural machine translation (nmt) models @xcite have shown promising quality. however, nmt often makes mistakes, especially on outliers such as out-of-domain samples, input noises, and lowfrequency words, on which models are potentially not trained or evaluated @xcite @xcite of their expertise and weaknesses, nmt lacks competency awareness, which limits the trustworthiness when it is widely adopted. consequently, it is necessary to equip nmt models with good awareness of their translation competency, i.e. the ability to give a translation along with the quality score of the translation. with good awareness of its competency, nmt can then take conservative actions (such as human intervention) on low-quality translations, and request fewer manual inspections for high-quality ones. in order to inform users of the competency of mt, a traditional alternative is to employ extra quality evaluation tools foot_0 that are independent of the mt model, including automatic evaluation metrics (metric; @xcite @xcite and quality estimation (qe; @xcite . as shown in in this paper, we propose a novel method, competency-aware neural machine translation (canmt), which has both translation and selfestimation capabilities without any ground-truth translation or expert-scored data for the competency assessment. based on the assumption that "a good translation should be able to reconstruct the meaning of source sentence from it", we extend the standard translation model with a self-estimator, which adopts a reconstruction strategy, and use the semantic gap between the original source sentence and its reconstruction to achieve competency awareness. our self-estimator is a part of the nmt model and exploits the continuous representations as input. the intuition we believe is that continuous representations are more informative about the decoding procedure of nmt, thus better reflecting the quality of translation candidates than the discrete output. moreover, we update the original nmt decoder to a two-stream decoder, in which translation and reconstruction information flows are separated to avoid the interference of the source sentence during reconstruction. we conduct experiments on four widely used translation tasks: cwmt17 chinese-to-english (@xmath0, aspec japanese-to-english (@xmath1, wmt14 french-to-english (@xmath2. results show that while keeping the same translation performance as transformer @xcite baseline, canmt exhibits promising capability on selfestimation of quality, which outperforms the stateof-the-art unsupervised metrics and qe methods such as bertscore @xcite , rtt @xcite and confidence-based qe @xcite . it is encouraging to see that our method even surpasses those existing supervised extra-estimation tools like bleurt @xcite and comet @xcite which depend on ground-truth translations or learn from expert-scored data. extensive analyses demonstrate the robustness of canmt that effectively tackles problems of out-of-domain, quality drift, and miscalibration in self-estimation. complementary effects are also observed between canmt and extra-estimation tools. ## competency-aware neural machine translation (canmt) given an arbitrary source sentence x, the goal of canmt model is to generate a target translation y and assign a self-estimated competency score q @xmath3, which can be represented as: we propose to exploit the reconstruction strategy and use the semantic gap between original source sentence and its reconstruction to enable competency awareness. the main challenge of this approach is how to capture as many quality issues of the predicted translation as possible in reconstructing, such as semantic or syntactic errors and disfluency. to overcome this challenge, we utilize the continuous representation from the nmt decoder, as it contains more information about the decoding procedure than a discrete translation and thus is easier to reconstruct the source sentence. ## experiments for the competency awareness of canmt, the evaluation of performance consists of two aspects: translation and self-estimation. we use bleu to validate translation performance. as for selfestimation, it considers whether an mt model is aware of its own competency, thus public metric and qe tests are unsuitable for us as the translations are from other models. therefore, we generate translations by canmt and evaluate the pearson correlations between the self-estimation scores and human judgments. ## analysis in this section, we conduct analyses on competency awareness to explain why canmt performs better. ## conclusion in this paper, we pay attention to the competency awareness of nmt. the major contributions of our work are three-fold: 1) we believe that competency awareness should be an essential capability of nmt and requires careful study. we hope our study can attract more interest in this problem; 2) we propose canmt, which is able to output both the competency and the translation for a given source sentence. it novelly leverages the internal information of itself to estimate the competency, showing superiority compared to other approaches that merely consider the outputs or the prediction distribution of nmt; and 3) empirical results exhibit the strong performance and robustness of canmt on quality estimation, which, to the best of our knowledge, is the first report that self-estimation method sur-passes supervised ones across translation directions and domains. for future work, it is a promising direction to employ competency awareness in reinforcement learning for nmt. the self-estimated competency scores correlate better with human judgments than traditional reinforcement learning rewards such as sentence-level bleu @xcite , we can use it to find weaknesses in nmt and take appropriate optimization.