BERTT une: Fine-Tuning Neural Machine Translation with BERTS core (2021)

Abstract: Neural machine translation models are often biased toward the limited translation references seen during training. To amend this form of overfitting, in this paper we propose fine-tuning the models with a novel training objective based on the recently-proposed BERTScore evaluation metric. BERTScore is a scoring function based on contextual embeddings that overcomes the typical limitations of n-gram-based metrics (e.g. synonyms, paraphrases), allowing translations that are different from the references, yet close in the contextual embedding space, to be treated as substantially correct. To be able to use BERTScore as a training objective, we propose three approaches for generating soft predictions, allowing the network to remain completely differentiable end-to-end. Experiments carried out over four, diverse language pairs show improvements of up to 0.58 pp (3.28%) in BLEU score and up to 0.76 pp (0.98%) in BERTScore (F_BERT) when fine-tuning a strong baseline.

## introduction neural machine translation (nmt) has imposed itself as the most performing approach for automatic translation in a large variety of cases @xcite . however, nmt models suffer from well-known limitations such as overfitting and moderate generalization, particularly when the training data are limited @xcite . this mainly stems from the fact that nmt models have large capacity and are usually trained to maximize the likelihood of just a single reference sentence per source sentence, thus ignoring possible variations within the translation (e.g. synonyms, paraphrases) and potentially resulting in overfitting. a somewhat analogous problem affects evaluation, where metrics such as bleu @xcite only consider as correct the predicted n-grams that match exactly in the ground-truth sentence. in order to alleviate the n-gram matching issue during evaluation, @xcite have recently proposed the bertscore metric that measures the accuracy of a translation model in a contextual embedding space. in bertscore, a pretrained language model (e.g. bert @xcite ) is first used to compute the contextual embeddings of the predicted sentence, ŷ1 , . . . , ŷk , and the reference sentence, y 1 , . . . , y l , with k and l word-pieces, respectively. then, recall (r bert ), precision (p bert ), and f1 (f bert ) scores are defined as cosine similarities between the normalized contextual embeddings. for example, the recall is defined as: in this work, we propose using bertscore as an objective function for model fine-tuning. our rationale is that bertscore is a sentence-level objective that may be able to refine the performance of nmt models trained with the conventional, tokenlevel log-likelihood. however, in order to fine-tune the model with bertscore as an objective, endto-end differentiability needs to be ensured. while the bertscore scoring function is based on word embeddings and is in itself differentiable, its input derives from categorical predictions (i.e. argmax or sampling), breaking the differentiability of the overall model. in this work, we solve this problem by generating soft predictions during training with three different approaches. one of the approaches, based on the gumbel-softmax @xcite , also leverages sampling, allowing the model to benefit from a certain degree of exploration. for immediacy, we refer to our approach as berttune. the experimental results over four, diverse language pairs have shown improvements of up to 0.58 pp (3.28%) in bleu score and up to 0.76 pp (0.98%) in bertscore with respect to a contemporary baseline @xcite . ## related work in recent years, various researchers have addressed the problem of overfitting in nmt models. this problem can be specially severe for neural models, given that, in principle, their large number of parameters could allow for a perfect memorization of the training set. for instance, @xcite have trained an nmt model using both a reference sentence and its bag-of-words vector as targets, assuming that the space of alternative, correct translations share similar bags-of-words. others @xcite have proposed smoothing the probability distribution generated by the decoder using the embedding distance between the predicted and target words, forcing the network to increase the probability of words other than the reference. another line of work has proposed to explicitly predict word embeddings, using the cosine similarity with the target embedding as the reward function @xcite . reinforcement learning-style training has also been used to alleviate overfitting @xcite . the use of beam search removes the exposure bias problem @xcite , and the use of sampling introduces some degree of exploration. in addition, these approaches allow using non-differentiable, sequencelevel metrics as reward functions. however, in practice, approximating the expectation of the objective function with only one or a few samples results in models with high variance and convergence issues. significant effort has also been recently dedicated to leveraging large, pretrained language models @xcite @xcite for improving the performance of nmt models. this includes using contextual word embeddings either as input features @xcite or for input augmentation @xcite , and using a pretrained language model for initializing the weights of the encoder @xcite . alternatively, @xcite have proposed using a pretrained language model as a prior, encouraging the network to generate probability distributions that have a high likelihood in the language model. in abstractive summarization, @xcite have used bertscore as reward in a deep reinforcement learning framework. in a similar vein, our work, too, aims to leverage pretrained language models for improving the nmt accuracy. however, to the best of our knowledge, ours is the first work to directly include a language model as a differentiable evaluation measure in the training objective. in this way, the nmt model is able to exploit the value of a pretrained language model while at the same time being fine-tuned over a task-specific evaluation metric. ## bertscore optimization translation evaluation metrics, including bertscore, typically require a predicted translation, ŷ1 , . . . , ŷk , and at least one reference translation, y 1 , . . . , y l , as inputs. at its turn, the predicted translation is typically obtained as a sequence of individual word (or token) predictions, using beam search or greedy decoding. we can express the predictions as: in brief, the computation of bertscore works as follows: the scorer first converts the words in the predicted and reference sentences to corresponding static (i.e., non-contextual) word embeddings using the embedding matrix, e, stored in the pretrained language model. for the predicted sequence, we note this lookup as: the sequences of static embeddings for the predicted and reference sentences are then used as inputs into the language model to generate corresponding sequences of contextualized embeddings, ŷ1 , . . . , ŷk and y 1 , . . . , y k , respectively, over which the bertscore is finally computed. for our work, we have chosen to optimize the f bert score as it balances precision and recall. for more details on the scoring function we refer the reader to @xcite . ## conclusion in this work, we have proposed fine-tuning nmt models with bertscore, a recently proposed word embedding-based evaluation metric aimed to overcome the typical limitations of n-gram matching. to be able to use bertscore as an objective function while keeping the model end-to-end differentiable, we have proposed generating soft predictions with differentiable operators such as the sparsemax and the gumbel-softmax. the experimental results over four language pairs have showed that the proposed approach -nicknamed berttune -has been able to achieve statistically significant improvements in bleu, f bert and ms scores over a strong baseline. as future work, we intend to explore the impact of key factors such as the dataset size, the sparsity degree of the predictions and the choice of different pretrained language models, and we also plan to evaluate the use of beam search/sequential sampling during training to leverage further exploration of candidate translations. this appendix provides detailed information about the preprocessing of the datasets and the hyperparameter selection to facilitate the reproducibility of the experiments. all the code will be released after the anonymity period. as part of the preprocessing of the datasets, all sentences have been tokenized and lowercased. the source languages have been tokenized with the moses tokenizer foot_1 , except for chinese that has been tokenized using jieba foot_2 . the target languages have instead been tokenized with the tokenizer learned by the pretrained language model. as language models for bertscore, we have used bert-base-uncased (en), dbmdz/ bert-base-turkish-uncased (tr) and dccuchile/bert-base-spanish-wwmuncased (es) from hugging face. as language model for the moverscore, we have used the suggested language model for english foot_3 , dbmdz/distilbert-base-turkishcased for turkish and mrm8488/distill-bert-base-spa nish -wwm-cased-fine tuned-spa-squad2-es for spanish, the last two from huggingface. the few sentences longer than 175 tokens have been removed from all datasets as in the original fairseq preprocessing script. additionally, further tokenization at subword level has been performed over the source languages using byte-pair encoding (bpe) @xcite with 32, 000 merge operations. an important step in the preprocessing has been to force the decoder and the language model to share the same vocabulary. therefore, we have assigned the decoder with the vocabulary from the selected pretrained language model, ensuring that both used identical bos, eos, pad and unk tokens. for training a strong transformer baseline, we have followed the recommendations in fairseq 7 . the architecture is the predefined transformer iwslt de en architecture (79m parameters) with word embedding and hidden vector dimension size of 512, and 6 transformer layers. we have set the training batch size to 4, 096 tokens, the dropout rate to 0.3 and the clip norm gradient clipping parameter to 0.0. the objective function is the label-smoothed negative log-likelihood, with the smoothing factor set to 0.1. we have used the adam optimizer (kingma and ba, 2015) with a 5e -4 learning rate and beta values β 1 = 0.9 and β 2 = 0.98. we have set the warm-up steps to 4, 000 with an initial learning rate of 1e -7. during training, we have reduced the learning rate with an inverse square-root scheduler and the weight decay set to 0.001. we have trained the model until convergence of the bleu score on the validation set, with checkpoints at each epoch and patience set to 3, or until the learning rate dropped below 1e -9. for fine-tuning with f bert , we have initialized the transformer models with the trained weights of the baseline. we have kept all hyperparameters identical, except for the learning rate which has been reduced by an order of magnitude to 5e -5, following common fine-tuning strategies. the models have been fine-tuned until convergence over the validation set, with patience set to 3. since the changes have only involved the training objective, the number of trainable parameters has remained exactly the same (79m). at test time, we have used beam search decoding with beam size 5 and length penalty 1. for all the experiments we have used an nvidia quadro p5000 gpu card with 16 gb of memory. in the example in finally, in the example in