Recovery Should Never Deviate from Ground Truth: Mitigating Exposure Bias in Neural Machine Translation (2024)

Abstract: In Neural Machine Translation, models are often trained with teacher forcing and suffer from exposure bias due to the discrepancy between training and inference. Current token-level solutions, such as scheduled sampling, aim to maximize the model’s capability to recover from errors. Their loss functions have a side effect: a sequence with errors may have a larger probability than the ground truth. The consequence is that the generated sequences may recover too much and deviate from the ground truth. This side effect is verified in our experiments. To address this issue, we propose using token-level contrastive learning to coordinate three training objectives: the usual MLE objective, an objective for recovery from errors, and a new objective to explicitly constrain the recovery in a scope that does not impact the ground truth. Our empirical analysis shows that this method effectively achieves these objectives in training and reduces the frequency with which the third objective is violated. We conduct experiments on three language pairs: German-English, Russian-English, and English-Russian. Results show that our method outperforms the vanilla Transformer and other methods addressing the exposure bias.

## introduction like many other text generation tasks, models for neural machine translation (nmt) @xcite are usually trained with teacher forcing. during training, ground truth tokens are used as target prefixes to the decoder, and the model learns to predict the next token conditioned on the ground truth. there is a discrepancy between this training method and inference. in inference, the ground truth tokens are not available. the target prefixes to the decoder are tokens previously generated by the model, which may include some errors. this discrepancy is referred to as exposure bias @xcite . the main concern about exposure bias is error accumulation. if one error happens at one step, it is incorporated into the future steps and leads to more errors. although there are still some doubts about whether exposure bias is a big issue for text generation @xcite , more research shows that this issue matters for nmt @xcite @xcite . there are two approaches to mitigate the exposure bias, working at the token and sequence levels, respectively. the token-level solutions, for example, scheduled sampling @xcite @xcite , usually use the tokens sampled from the model to replace the ground truth in training. the objective is to simulate the possible errors in inference and recover from these errors to reduce the error accumulation. the sequence-level solutions directly maximize the total quality of the generated sequences with a sequence-level loss function @xcite @xcite . there is still debate whether these solutions are stable and effective @xcite . this paper focuses on mitigating the exposure bias with token-level objectives. the loss functions used in most token-level solutions have a side effect. they aim to increase the model's capability to recover from errors by maximizing the probability of the next token conditioned on some error tokens. consequently, a sequence with errors may have a larger probability than the ground truth, and the generated sequences may deviate from the ground truth. this side effect is verified in our experiments. we discover a missing objective behind this side effect that can explicitly constrain the recovery in a scope that does not impact the ground truth. we propose to use token-level contrastive learning and coordinate three training objectives: the usual maximum likelihood estimation (mle) objective, an objective for recovery from errors, and a new objective constraining the recovery. our empirical analysis shows that this method effectively meets three objectives in training. particularly our method reduces the frequency that the third objective is violated. we conduct experiments on german-english (de-en), russian-english (ru-en), and english-russian (en-ru). results show that our method outperforms the vanilla transformer and other methods addressing the exposure bias. ## analysis besides the overall performance, we investigate how these three objectives are met and whether the loss function effectively coordinates these objectives. we start by using the prefix switching method. then, we directly measure how often the three objectives in subsection 3.1 are not met during decoding for each method. finally, we verify the effectiveness of the loss function in subsection 3.3 by monitoring how the values of these components in our loss function change during training tcl and its variants. ## conclusion current token-level methods addressing exposure bias may have a side effect: a sequence with errors may have a larger probability than the ground truth. consequently, the generated sequence may deviate from the ground truth. our experiments verify this side effect. we discover a missing objective behind this side effect that can explicitly constrain the recovery in a scope that does not im-pact the ground truth. we propose token-level contrastive learning to coordinate three objectives in the loss function: the original mle, recovery from errors, and constraining the recovery in a scope not to exceed the ground truth. experimental results on three language pairs show that our method outperforms the vanilla transformer and five methods aiming at mitigating exposure bias. empirical analysis demonstrates that this method achieves a pareto optimality compared with the vanilla transformer. it is also verified that each component in our loss function effectively improves the model in training.