Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation (2023)

Abstract: The decoder in simultaneous neural machine translation receives limited information from the source while having to balance the opposing requirements of latency versus translation quality. In this paper, we use an auxiliary target-side language model to augment the training of the decoder model. Under this notion of target adaptive training, generating rare or difficult tokens is rewarded which improves the translation quality while reducing latency. The predictions made by a language model in the decoder are combined with the traditional cross entropy loss which frees up the focus on the source side context. Our experimental results over multiple language pairs show that compared to previous state of the art methods in simultaneous translation, we can use an augmented target side context to improve BLEU scores significantly. We show improvements over the state of the art in the low latency range with lower average lagging values (faster output).

## introduction simultaneous machine translation (simt; @xcite ; @xcite ) is a special case of neural machine translation (nmt; @xcite ) that aims to produce real time-translations in the target language from a streaming input in the source language. the cornerstone of this task, as well as a key challenge, is the trade-off between the translation quality and the latency in producing the translations. this balance is ensured by a fixed @xcite or adaptive @xcite @xcite read/write policy that determines whether to wait for the next source token (a read action) or to generate a translation (a write action). adaptive policies dynamically predict the action based on 1 github.com/sfu-natlang/target_rescale_simt * equal contribution. listing order is random. work performed while aj was visiting sfu natlang lab. it 's quite warm in winter , but summer is extremely hot . <@xmath0.0 0.1 0.2 it was cozy in winter but extremely hot in summer . lm mma reads 0 1 2 3 4 1 0.7 1.2 simt policy source target conditional max. likelihood language model target context token importance 1 1 1 rescaling ours prior work figure 1 : prior work on simultaneous mt weighs every target token equally. top: normalized negative log-likelihood (nll) scores of each generated in-context target token as scored by the baseline simt along with the number of reads preceding a target token, and a target language model (lm). as the translations are imperfect, the lm shows disagreement by following an opposite nll trend compared to the translation model. bottom: our method rescales the importance of each target token using the target context during training. the current source and target contexts @xcite . although adaptive policies achieve a better latency/bleu trade-off, they often fail to account for the varying importance of different tokens when deciding a read/write action. in figure 1 @xcite , there is a negative correlation between the normalized negative log likelihoods of output tokens as measured by mma (a simt model with an adaptive policy; @xcite versus a left-to-right language model (lm). this reflects a translation which the simt model is confident about, but which the lm regards as poor english (possibly due to the semantic mismatch evident in "warm in winter"). since a simultaneous policy can only access partial source context, its outputs are likely to reflect imperfect guesses such as these, particularly when translating in realtime between language pairs with different word orderings (subject-object-verb) and very long compounds. as a result, training objectives which treat all translated tokens with equal importance are suboptimal. in the context of translation, content words are generally considered more informative than function words @xcite . this is because content words carry the main semantic and lexical meaning of a sentence, while function words provide grammatical context and help to convey the syntactic structure of a sentence. similarly, highfrequency words that are easier for the translation model to generate may sometimes carry less information than the desirable low-frequency (rare) words that the model struggles to generate @xcite . to this end, @xcite proposed to leverage conditional mutual information (mi) to estimate the weight-coefficients between the source and target to reweigh the importance of each target token. however, such an approach hasn't been explored to address simultaneous or streaming mt to the best of our knowledge, and the lack of a complete source context makes the adaptation of this method to simt non-trivial. to improve simultaneous mt, @xcite proposed a prediction mechanism on the source side to get future information and aid the lack of information on target-side for translation. instead of directly predicting a source token, @xcite predict its aligned future-position for a given target token to guide its policy. on the other hand, @xcite and @xcite explored policies that assign varying importance to source/target tokens based on their level of information, with more informative tokens having a greater influence on the model. in this paper, we propose a technique to alleviate this problem in simt using an information theoretic approach and an adaptive training paradigm. inspired by the recent work in using pointwise mutual information for guiding the decoder in fullsentence (non-simultaneous) translation @xcite , we differentiate the importance of various target tokens by their dependence on the source sentence. as shown in figure 1 (bottom), to guide our simultaneous translation model, we incorporate a language model that provides an additional signal indicating the importance of each target token or sentence. this target-context aware estimation leverages the relative probabilities of the translation model and language model to guide the generation process by explicitly re-weighting the training loss of each target token in the translation. experiments show the strength of our simple method, outperforming several strong baselines in terms of both latency and bleu scores. we perform exhaustive analysis to show that our model performs particularly well on translating low frequency words and longer sentences. frequency based approaches @xcite to assign these weights are promising but maintaining a frequency count can be an expensive overhead and would not be directly transferable to a simultaneous setting. more recently, @xcite proposed to leverage pointwise mutual information (mi) to estimate the weight-coefficients between the source x and target y as : mi(x, y) = log p(x, y) p(x).p(y) ## results figure 2 shows the comparison of bleu vs. latency (in terms of average lagging) of our method against previous methods on the iwslt'15 @xmath1. for @xmath2, we observe a significant improvement in the bleu scores at the same latencies, compared to the baselines. we also reach the offline translation quality in low al on this dataset. in the @xmath3, @xmath4, there is a boost in the translation quality, more noticeably for lower latencies. the plots show that our method boosts translation quality in the earlier latencies and the effect of reweighing is more pronounced in these regions, where the source context is more limited. in higher latency regions, when the source information window increases, the other baselines start to reach our bleu score in the english-german directions. in figure 3 , we compare against several stateof-the-art methods on the @xmath5. our method gets better translation quality compared all others, in the low-latency zone, matching the offline score at 3.86 al. we show the bleu vs. al plot in a low latency range to compare performance in the more challenging area of this task, the low latency points. ## conclusion we have presented a simple technique for rescaling target-token importance in simultaneous translation using an information theoretic approach and an adaptive training paradigm. we differentiate the importance of various target tokens by their dependence on the source sentence. to guide our simultaneous translation model, we incorporate a targetside language model that provides an additional signal indicating the importance of each target token or sentence under the condition of the previous target context. our model shows strong performance on several datasets and outperforms several state-ofthe-art techniques in the low latency range (@xmath6. further analysis shows that our technique is better able to translate long sentences and those with rare words. we also showed that the translation path (read/write action sequence) has a stronger correlation to the source-target alignment.