Context Consistency between Training and Inference in Simultaneous Machine Translation (2024)

Abstract: Simultaneous Machine Translation (SiMT) aims to yield a real-time partial translation with a monotonically growing source-side context.However, there is a counterintuitive phenomenon about the context usage between training and inference: *e.g.*, in wait- k inference, model consistently trained with wait- k is much worse than that model inconsistently trained with wait- k' ( k'\neq k ) in terms of translation quality. To this end, we first investigate the underlying reasons behind this phenomenon and uncover the following two factors: 1) the limited correlation between translation quality and training loss; 2) exposure bias between training and inference. Based on both reasons, we then propose an effective training approach called context consistency training accordingly, which encourages consistent context usage between training and inference by optimizing translation quality and latency as bi-objectives and exposing the predictions to the model during the training. The experiments on three language pairs demonstrate that our SiMT system encouraging context consistency outperforms existing SiMT systems with context inconsistency for the first time.

## introduction simultaneous machine translation (simt) @xcite @xcite aims to generate a partial translation while incrementally receiving a prefix of a source sentence. simt plays an very improtant role in many real-world scenarios such as multilateral organizations and international summits @xcite . a good simt system should not only have low latency in the generation process but also yield a translation with high quality. hence, there has recently been witnessed a surge of interest in the bleu 19.10 training k' = 1 1 ... ... ... ... 9 training k' = 9 ... ... ... ... inference @xmath0... ... ... ... ctx cons ctx incons bleu 21.42 figure 1 : counterintuitive phenomenon on the context usage between training and inference: in wait-1 inference (@xmath1, model trained with k ′ =9 (denoted by "ctx incons") outperforms the model trained with k ′ =1 (denoted by "ctx cons") in terms of bleu, even though the former model (trained by k ′ =9) induces a mismatch on context usage between training and inference. in this paper, we shed light on a counterintuitive phenomenon on the context usage between training and inference in simt: in wait-k inference, model consistently trained with wait-k is worse than that model inconsistently trained with waitk ′ (k ′ ̸ =k) in terms of bleu scores @xcite , as shown in figure 1 . this phenomenon was first observed by @xcite yet without explanations. subsequently, such context inconsistency training becomes a standard practice @xcite @xcite , even if this phenomenon is counterintuitive due to the mismatch between training and inference on the usage of partial source-side context. to investigate the reasons behind the above counterintuitive phenomenon, we conduct experiments from two perspectives: calculating the correlation between bleu scores and cross-entropy loss, as well as evaluating the translation quality under the prefix-constrained decoding setting. our empirical experiments demonstrate two reasons that are responsible for the phenomenon: 1) the limited correlation between translation quality and training loss; 2) exposure bias between training and inference. moreover, based on our findings, this paper proposes an effective training approach called context consistency training. its key idea is to make the context usage consistent between training and inference by optimizing translation quality and latency as bi-objectives and exposing the model to its own predictions during the training stage. particularly, this approach is general to be applied to most simt systems. experiments conducted across various benchmarks demonstrate that the proposed context consistency training towards bi-objectives achieves substantial gains over the original consistency training based on cross-entropy. our main contributions are: ## context consistency training for simt previous findings have shown that: 1) it is helpful to address the counterintuitive phenomenon by encouraging the consistent criterion between training and inference; 2) exposure bias is a reason for the counterintuitive phenomenon. to address the counterintuitive phenomenon and make the consistent model successful, we propose a simple yet effective training approach, called context consistency training for simt, which not only incorporates the evaluation metric (e.g., bleu, comet and chrf) for simt as training objectives ( §3.1) but also allows the model to expose its predictions during training ( §3.2). ## related work existing simt sudies can be mainly categorized into two types (i.e., fixed or adaptive policy) according the read/write policy. as the fixed policy, @xcite ground truth target "and she 's an optometrist in st. petersburg , and she plays with optics ." consistency-ce(wait-3) hypothesis "and it 's an optical optical in st. petersburg , and it 's playing with optics ." inconsistency-ce(wait-9) hypothesis "and it 's an optimist in st. petersburg , and it plays with optics ." consistency-bi(wait-3) hypothesis "and she 's an optic woman st. petersburg , and she plays with optics ." the above studies overlook the counterintuitive phenomenon about the context usage between training and inference, and our work thereby provides comprehensive analysis on this phenomenon and propose an effective approach to address this phenomenon, which is general enough to be applied into both policies. ## conclusion this paper pays attention to a counterintuitive phenomenon in the context of usage between training and inference in simt. subsequently, we conduct a comprehensive analysis and make the noteworthy discovery that this phenomenon primarily stems from the weak correlation between translation quality and training loss as well as exposure bias between training and inference. based on our findings, we propose a context consistency training method that incorporates both translation quality and latency as bi-objectives and alleviates the exposure bias issue during the training. experiments verify the effectiveness of the proposed approach, making the contextconsistent simt successful for the first time.