Using Context in Neural Machine Translation Training Objectives (2020)

Abstract: We present Neural Machine Translation (NMT) training using document-level metrics with batch-level documents. Previous sequence-objective approaches to NMT training focus exclusively on sentence-level metrics like sentence BLEU which do not correspond to the desired evaluation metric, typically document BLEU. Meanwhile research into document-level NMT training focuses on data or model architecture rather than training procedure. We find that each of these lines of research has a clear space in it for the other, and propose merging them with a scheme that allows a document-level evaluation metric to be used in the NMT training objective. We first sample pseudo-documents from sentence samples. We then approximate the expected document BLEU gradient with Monte Carlo sampling for use as a cost function in Minimum Risk Training (MRT). This two-level sampling procedure gives NMT performance gains over sequence MRT and maximum-likelihood training. We demonstrate that training is more robust for document-level metrics than with sequence metrics. We further demonstrate improvements on NMT with TER and Grammatical Error Correction (GEC) using GLEU, both metrics used at the document level for evaluations.

## introduction neural machine translation (nmt) research has explored token-level likelihood functions @xcite and sequence-level objectives inspired by reinforcement learning @xcite or expected minimum risk training (mrt) @xcite . a typical sequence objective in these cases is based on sentence-level bleu (sbleu) @xcite . however * now at google sbleu, even if aggregated over sentences, is only an approximation of the desired metric, documentlevel bleu. beyond translation, many metrics for natural language tasks do not have robust sentencelevel approximations. a logical progression is the extension of sequence-level nmt training objectives to include context from outside the sentence. document-based nmt, by contrast, aims to use out-of-sentence context to improve translation. recent research explores lexical consistency by providing additional sentences during training @xcite @xcite or inference @xcite , potentially with adjustments to model architecture. however, to the best of our knowledge, no attempt has been made to extend sequence-level neural training objectives to include document-level reward functions. this is despite document-level bleu being arguably the most common nmt metric, and being the function originally optimised by minimum error rate training (mert) for statistical machine translation (smt) @xcite . we propose merging lines of research on training objectives and document-level translation. we achieve this by presenting a document-level approach to sequence-level objectives which brings the training objective closer to the actual evaluation metric, using mrt as a representative example. we demonstrate mrt under document-level bleu as well as translation edit rate (ter) @xcite , which while decomposable to sentence level is less noisy when used over documents. we consider both pseudo-documents where sentences are assigned randomly to a mini-batch, and true document context where all sentences in the batch are from the same document. we finally apply our scheme to supervised grammatical error correction, for which using neural models is becoming increasingly popular @xcite @xcite . we show gains in gec metrics gleu @xcite and m2 @xcite . ## experiments we report on english-german nmt. we initialize with a baseline trained on 17.5m sentence pairs from wmt19 news task datasets @xcite , on which we learn a 32k-merge joint bpe vocabulary @xcite . we validate on newstest2017, and evaluate on newstest2018. we apply mrt only during fine-tuning, following previous work @xcite . in early experiments, we found that training from scratch with discriminative objectives (sequence-or document-based) is ineffective. we suspect samples produced early in training are so unlike the references that the model never receives a strong enough signal for effective training. we fine-tune on old wmt news task test sets @xcite @xcite @xcite @xcite @xcite in two settings. with random batches sentences from different documents are shuffled randomly into mini-batches. in this case doc-mrt metrics are over pseudo-documents. with document batches each batch contains only sentences from one document, and doc-mrt uses true document context. we use the same sampling temperatures and the same risk sharpness factors for both forms of mrt for each experiment. for grammatical error correction (gec) we train on sentences from nucle @xcite and lang-8 learner english @xcite with at least one correction, a total of 660k sentences. we evaluate on the jfleg @xcite for all models we use a transformer model @xcite with the 'base' ten-sor2tensor parameters @xcite . we train to validation set bleu convergence on a single gpu. the batch size for baselines and mle is 4096 tokens. for mrt, where each sentence in the batch is sampled n times, we reduce batch size by n while delaying gradient updates by the same factor to keep the effective batch size constant @xcite . at inference time we decode using beam size 4. all bleu scores are for cased, detokenized output, calculated using sacrebleu @xcite . ## conclusions and future work we present a novel approach for structured loss training with document-level objective functions. our approach relies on a procedure for sampling a set of diverse batch-level contexts using n-wise sample ordering. as well as randomly selecting training data, we assess training with mini-batches consisting only of single document contexts. while the scope of this work does not extend to sampling sentences given document context, this would be an interesting direction for future work. we demonstrate improvements covering three document-level evaluation metrics: bleu and ter for nmt and gleu for gec. we finish by noting that the original mert procedure developed for smt optimised document-level bleu and with our procedure we reintroduce this to nmt.