Reducing Length Bias in Scoring Neural Machine Translation via a Causal Inference Method (2021)

Abstract: Neural machine translation (NMT) usually employs beam search to expand the searching spaceand obtain more translation candidates. However the increase of the beam size often suffersfrom plenty of short translations resulting in dramatical decrease in translation quality. In this paper we handle the length bias problem through a perspective of causal inference. Specially we regard the model generated translation score S as a degraded true translation quality affectedby some noise and one of the confounders is the translation length. We apply a Half-Sibling Re-gression method to remove the length effect on S and then we can obtain a debiased translation score without length information. The proposed method is model agnostic and unsupervised which is adaptive to any NMT model and test dataset. We conduct the experiments on three translation tasks with different scales of datasets. Experimental results and further analyses showthat our approaches gain comparable performance with the empirical baseline methods.

## introduction recently, with the renaissance of deep learning, end-to-end neural machine translation (nmt) @xcite has gained remarkable performances @xcite @xcite . nmt models are usually built upon an encoder-decoder framework @xcite : the encoder reads an input sequence @xmath0, ..., x tx } into a hidden memory h, and the decoder is designed to model a probability over the translation @xmath1, ..., y t ŷ } by: the second strategy, the correcting ratio γ of the reward is usually determined by supervised training @xcite or manually fine-tuning @xcite before the testing stage, which lacks the ability of self-adapting to the unseen data. in this paper, we introduce a causal motivated model agnostic and unsupervised method to solve the length bias problem for nmt. as shown in fig. 1 , for a translation hypothesis ŷ, suppose that q is an unobservable true translation quality of ŷ, and the model generated score s can be seen as an observed degraded version of q which is affected by some noise n . generally, s equals s(ŷ) in conventional nmt approaches, and it can be viewed as one of the measurement methods of q with systematic errors. as mentioned above, one kind of systematic errors has a strong correlation with the translation length, therefore, the noise caused by length will be eliminated if we subtract the length effect from s. specifically, we utilize the half-sibling regression (hsr) @xcite method to perform the noise elimination operation for nmt. the method first apply a regression model to appraise the effect of the translation length on the model generated score, i.e. e[s|t ŷ]. then, the denoised score is obtained by removing e[s|t ŷ] from s: s := s -e[s|t ŷ]. we propose two branches of the framework, corpus based (c-hsr) and single source sentence based (s-hsr) re-scoring method. the difference is that c-hsr performs the estimation of e[s|t ŷ] on the whole test set, while s-hsr uses the translation candidates in a beam of the nmt inference process to predict e[s|t ŷ]. the operation of approximating e[s|t ŷ] for both c-hsr and s-hsr entirely rely on the current testing data of nmt without fine-tuning or any supervised information. in this work, we regard the nmt model as a black-box and apply the hsr-based denoised method to the re-scoring procedure for nmt. we conduct the experiments on three translation tasks: @xmath2, @xmath3, which represent low-resource, medium-resource and high-resource nmt, respectively. the experimental results show the proposed approaches achieve comparable performances with empirical length normalization methods. further analyses show the flexibility of the proposed methods and the assumptions that our approaches rely on are reliable. ## related work the length bias reduction methods can be mainly divided into two categories: i) dividing the log probability by the length penalty lp: for the first branch, the predominant form of the length penalty lp is the length of the hypothesis @xcite @xcite c c l 2 0 2 1 2020). google's nmt system @xcite employ an empirical length penalty that is computed as: the second branch is similar to the word penalty in statistical machine translation @xcite . the parameter γ can be automatically optimized with supervised learning @xcite or manually assignment @xcite . @xcite propose a log-linear nmt framework which incorporates a word reward feature to the framework to control the length of the translation: the proposed hsr-based debiasing method is motivated entirely by a causal structure shown in fig. 1 , although the form of the approach is same as the reward-based length normalization in eq. ( 5 ). formally, we can regard e[s|t ŷ] in eq. ( 3 ) as an instance of (γ • r) in eq. ( 9 ) with very few prior assumptions or handcrafted designs. the leaning process of e[s|t ŷ] is entirely model agnostic and unsupervised, which makes the proposed method more competitive to the previous supervised approaches @xcite @xcite in real practical applications. ## conclusion and future work in this paper, we introduce a causal motivated method to reduce the length bias problem in nmt. we employ a half-sibling regression @xcite ) method to handle this task and corroborate the task satisfies the independence assumption of hsr. experimental results on three language pairs with distinct data scales show the effectiveness of the proposed method. in the future, we will complete our experiments on the task of quality estimation. since the proposed approaches are model agnostic and unsupervised, we will verify the effectiveness of our approaches on other natural language generation tasks, such as dialogue system and summarization. we also hope to make further studies about the relationships among q, s and t y .