Contrastive Preference Learning for Neural Machine Translation (2024)

Abstract: There exists a discrepancy between the token-level objective during training and the overall sequence-level quality that is expected from the model. This discrepancy leads to issues like exposure bias.To align the model with human expectations, sequence-level objectives are often used to fine-tune pre-trained models.In this paper, we introduce a contrastive preference model that enhances the traditional Plackett-Luce model by incorporating an indicator function. Building upon this novel preference model, we propose Contrastive Preference Learning (CPL), which uses offline samples with list-wise preferences to fine-tune a pre-trained model in Neural Machine Translation. Our experiments, conducted on three language pairs, demonstrate that CPL outperforms not only the vanilla Transformer model but also other token-level and sequence-level baselines. Furthermore, the ablation study highlights the essential role of the proposed indicator function in achieving this improvement.

## introduction neural machine translation (nmt) models @xcite , like many other text generation tasks, are typically trained using teacher forcing and the token-level maximum likelihood estimation (mle) as the objective function. however, there exists a discrepancy between this training approach and the actual goal of a sequence generation system, which is to improve sequencelevel quality as measured by evaluation metrics like bleu, or human evaluation. one issue stemming from this disparity is exposure bias @xcite @xcite . @xcite also link this discrepancy to other issues observed in nmt: hallucination, domain shift, and beam search curse @xcite . this same discrepancy is the underlying reason behind the topic of alignment in large language models (llms). aligning llms with human expectations has been recognized as an important objective for future artificial general intelligence (agi) foot_0 , leading to an active research area @xcite . approaches developed in both domains can be mutually beneficial. to mitigate this discrepancy, sequence-level objectives are often used to fine-tune a pre-trained model @xcite . there are two lines of related research: reinforcement learning (rl) with online samples and supervised learning with offline samples. in the approach using rl with online samples, samples are refreshed by drawing from the model at every training step. this approach has been extensively discussed in nmt. mixer @xcite and minimum risk training (mrt) @xcite are two prominent implementations. however, there is still ongoing debate regarding the stability and effectiveness of these solutions @xcite . in llms, the use of reinforcement learning with human feedback (rlhf) @xcite is common. the reward function for rl is trained using offline preference samples labeled by human experts, but online samples are still used during the rl phase. methods utilizing online samples are significantly slower than offline ones. the alternative approach is supervised learning with offline samples. these samples are drawn from the pre-trained model once and then ranked and used as training data to fine-tune the model with supervised learning. unlike online samples, these offline samples are not refreshed during training. this method has been explored in text summarization through contrastive learning with list-wise ranking @xcite @xcite . in llms, direct preference optimization (dpo) foot_1 @xcite uses a loss function which has been theoretically proven equivalent to rl with preference data. however, their experiments primarily focus on pair-wise preference rather than list-wise ranking. our proposal, contrastive preference learning (cpl), follows the second approach since it is more efficient and stable compared to the rl with online samples. we begin by augmenting the classic list-wise plackett-luce (pl) preference model @xcite with an indicator function. this indicator function incorporates a constraint commonly used in contrastive learning to prevent overfitting. then, the training objective is derived by applying the reward function in dpo to this augmented pl model. our experiments consider three language pairs and compare cpl against various baselines. these baselines include the vanilla transformer, contrastive learning and dpo with offline list-wise ranking, two online sequence-level methods (mixer and mrt), and three token-level methods aimed at mitigating the exposure bias. the results show that cpl significantly outperforms the vanilla transformer and achieves the best performance among all methods. the ablation study shows the crucial role played by the proposed indicator function. ## conclusion using the sequence-level objective to fine-tune a pre-trained model is a promising way to align the model, trained with a token-level objective, with human expectations for high sequence-level quality. we augment the classic plackett-luce model with an indicator function. based on this novel contrastive preference model, we propose contrastive preference learning (cpl), which uses offline samples with list-wise preference to finetune a pre-trained model. our experiments on three language pairs demonstrate that cpl outperforms the vanilla transformer model and other token-level and sequence-level baselines. the proposed indicator function applies a constraint used in contrastive learning to prevent overfitting. its crucial role is demonstrated in our ablation study.