S im CLNMT : A Simple Contrastive Learning Method for Enhancing Neural Machine Translation Quality (2024)

Abstract: “Neural Machine Translation (NMT) models are typically trained using Maximum LikelihoodEstimation (MLE). However, this approach has a limitation: while it might select the bestword for the immediate context, it does not generally optimize for the entire sentence. Tomitigate this issue, we propose a simple yet effective training method called SimCLNMT.This method is designed to select words that fit well in the immediate context and also en-hance the overall translation quality over time. During training, SimCLNMT scores multiplesystem-generated (candidate) translations using the logarithm of conditional probabilities.Itthen employs a ranking loss function to learn and adjust these probabilities to align with thecorresponding quality scores. Our experimental results demonstrate that SimCLNMT consis-tently outperforms traditional MLE training on both the NIST English-Chinese and WMT’14English-German datasets. Further analysis also indicates that the translations generated by ourmodel are more closely aligned with the corresponding quality scores. We release our code athttps://github.com/chaos130/fairseq_SimCLNMT.Introduction”

## introduction neural machine translation (nmt) @xcite @xcite is a pivotal task within the field of natural language processing. it focuses on translating text from one language (source) to another (target) using end-to-end neural network models. this task has garnered increasing attention in both academia and industry @xcite @xcite . notably, the emergence of the transformer architecture @xcite marks a paradigm shift in nmt. characterized by its innovative self-attention mechanism, the transformer enables the model to variably prioritize the significance of words within the input sentence. this results in more effectively capturing the context compared to previous models, such as the recurrent neural network (rnn) @xcite and convolutional neural network (cnn) @xcite . despite the advancements of the transformer, it still relies on the maximum likelihood estimation (mle) for word prediction at each time step. while this practice optimizes for immediate word selection probability, it may not consistently enhance the overall translation quality. this discrepancy between probabilistic optimality and actual translation quality highlights a critical limitation in current nmt approaches. to assess the practical impact of this limitation, we conducted a preliminary study using a pre-trained nmt model (mle baseline). our process involves generating two candidate translations for each input and analyzing whether higher probability correlates with higher bleu scores @xcite . as depicted in as suggested by @xcite , one primary strategy to mitigate the mentioned issues involves adopting reinforcement learning (rl). rl training allows models to be fine-tuned with reward mechanisms that assess the overall quality of predictions. this technique focuses on the result of the translation process rather than just the immediate predictions of the next word. however, it introduces common challenges to deep rl, including noisy gradient estimation @xcite , which can result in unstable training and high sensitivity of hyperparameter settings. moura ramos et al. (2023) further explores this concept by incorporating evaluation metrics learned from human quality annotations as reward models within the nmt framework. this approach enhances translation quality through data filtering, rl training, and re-ranking techniques. as an alternative strategy, minimum risk training (mrt) has been adopted for language generation tasks to overcome the mle training challenge @xcite . despite its potential, the accuracy of the mrt is limited by the number of output samples. @xcite introduces a novel method leveraging generative adversarial network (gan) for nmt, aiming to train the model to generate sentences indistinguishable from human translations. additionally, @xcite and @xcite have demonstrated that contrastive learning techniques can effectively improve the quality of model output. in this paper, we seek to address the mentioned issues by incorporating contrastive learning into the nmt framework. inspired by @xcite and @xcite , we introduce simclnmt, a simple yet effective approach that aligns the probabilities of various candidate translations with their corresponding quality scores via a ranking loss mechanism. our methodology is divided into two phases: pre-training and fine-tuning. during the pre-training phase, we train a nmt model using a comprehensive data set, which serves as a strong mle baseline for the fine-tuning stage. in the fine-tuning phase, we generate several candidate translations from the mle baseline model. each candidate is then evaluated using an automated metric to determine its quality score, yielding pairs of candidate translations and quality scores for every source sentence. simclnmt computes the log probability for each candidate and uses a ranking loss mechanism to align these probabilities with their corresponding quality scores, ensuring that the model predictions are consistent with the evaluations provided by the metric. in summary, we mainly make the following contributions: the training of nmt models typically relies on mle. while mle excels at selecting contextually appropriate words, it often falls short of optimizing overall sentence performance, which can result in suboptimal translation outputs. several previous works have aimed to mitigate this issue. @xcite explores the use of reinforcement learning (rl) to improve machine translation with human feedback. this approach fine-tunes nmt models using rl techniques and rewards that assess the overall c c l 2 0 2 4 quality of translations, rather than focusing solely on predicting the next word. however, it also faces the inherent challenges of deep rl, including the problem of noisy gradient estimation. moura ramos et al. (2023) extends the exploration of using rl for nmt by integrating evaluation metrics derived from human feedback as reward models within the nmt framework. this methodology encompasses multiple strategies that include data filtering to refine the training corpus, rl training to adjust the nmt model based on rewards reflecting translation quality, and re-ranking techniques at inference time. this comprehensive approach seeks to enhance translation quality by ensuring that the outputs of the nmt model are more closely aligned with human judgments. @xcite presents a significant shift in nmt training methodologies by incorporating evaluation metrics as loss functions. this allows the model to learn patterns that better align with the quality of the final translation output, rather than just the likelihood of word sequences. @xcite introduces a novel method that utilizes gan for nmt. in this approach, the nmt model acts as the generator, producing human-like translations, while a discriminator is trained to distinguish between machine and human translations. the goal is to iteratively improve the nmt model based on discriminator feedback, training it to generate translations indistinguishable from human ones and directly optimizing the output quality. in addition to the above methodologies, contrastive learning has gained traction, particularly in abstractive summarization and large language models. @xcite and @xcite introduce contrastive learning to the domain of abstractive summarization, focusing on learning from higher-quality sentences rather than those with high model probabilities. this approach shifts the focus towards generating summaries that are statistically likely and of higher relevance and quality. @xcite further refines the integration of human feedback into language model fine-tuning through a novel training paradigm called rrhf. this method evaluates sampled responses from various sources using the logarithm of conditional probabilities and employs a ranking loss mechanism to align these probabilities with human preferences. ## simclnmt this section breaks down the holistic training process into two stages: pretraining (section 3.1) and finetuning (section 3.2). ## limitation and future work