The D eep M ind C hinese– E nglish Document Translation System at WMT 2020 (2020)

Abstract: This paper describes the DeepMind submission to the Chinese \rightarrow English constrained data track of the WMT2020 Shared Task on News Translation. The submission employs a noisy channel factorization as the backbone of a document translation system. This approach allows the flexible combination of a number of independent component models which are further augmented with back-translation, distillation, fine-tuning with in-domain data, Monte-Carlo Tree Search decoding, and improved uncertainty estimation. In order to address persistent issues with the premature truncation of long sequences we included specialized length models and sentence segmentation techniques. Our final system provides a 9.9 BLEU points improvement over a baseline Transformer on our test set (newstest 2019).

## introduction the wmt2020 shared task on translating news data from chinese into english provides a challenging test for machine translation systems and an ideal domain for researchers to evaluate new techniques. the deepmind submission to the constrained data track is based on the modular noisy channel document translation architecture advocated by @xcite . in this formulation, the posterior probability of a translation is the product of the unconditional probability of the output document (the language model) and the conditional probability of the translation from the output to source (the channel model). by assuming sentences within a document are independently translated, we can train the channel model using readily available parallel sentences, rather than being reliant on less numerous parallel documents, and the language model on monolingual documents. this modular approach allows the components of the system to be implemented and optimized independently while at inference time, when we reason over the posterior distribution of translations given the source document, conditional dependencies between translations are induced by the language model prior. the core of our document-level translation architecture is the noisy channel reranker. it requires proposal, channel, and language models, each of which is optimized separately using different techniques and approaches. for the proposal and channel models we use transformer models @xcite ( §4.1) with data augmentation ( §4.2), such as back translation @xcite , distillation @xcite , and forward-translated parallel documents. we further improve these sequence-to-sequence (seq2seq) models by fine-tuning them with in-domain data ( §4.3). to improve the robustness of the reranker we apply adversarial training and contrastive learning methods for uncertainty estimation ( §4.4). finally, we include candidate translations generated by monte-carlo tree search (mcts) ( §b) in order to improve the diversity of the candidate pool for the reranker. our language models are based on the transformer-xl architecture @xcite and optimized with distillation and fine-tuning with in-domain data ( §5). during development, we observed weaknesses in our system's translations for long sentences, largely due to premature truncations. we developed several techniques to mitigate this issue such as sentence segmentation (breaking sentences into logical complete segments) and training specialized models with synthetically constructed long sequences to generate additional proposals for our reranker @xcite . experiments show that the aforementioned techniques are very effective: our system outperforms the transformer baseline by 9.9 bleu points on our test set (newstest2019). our final system achieves a bleu score of 35.4 on the @xmath0. ## document translation via bayes' rule following @xcite , we model document translation via bayes' rule. we define @xmath1, x 2 , . . . , x i ) as the source document with i sentences, and similarly, @xmath2, y 2 , . . . , y j ) as the target document with j sentences, where x i and y j denote the ith sentence in the source document and the jth sentence in the target document respectively. we assume that @xmath3. the translation of a document x is determined by finding the document ŷ , where p( ŷ | x) is maximal. we further assume that sentences are independently translated, and that the sentences within a document admit a left-to-right factorization according to the chain rule. therefore, we have the advantages of this formulation is that during training the translation models can be learned from parallel sentences and monolingual documents which are vastly available in practice unlike parallel documents. during test time, when a source document is observed, conditional dependencies between the translation of the source sentences are created in the posterior. ## training data to train all of the models used in our system, we made use only of the constrained data provided to shared task participants. in this section, we discuss the preprocessing and normalization techniques we carried out in an attempt to reduce spurious uncertainty in the modeling problem. text preprocessing we carried out the following text normalization steps prior to use in any models: ## proposal and channel models the proposal model, used to generate candidate translations, and the scoring models (proposal probability model, adversarially-trained proposal model, channel model), used to compute features for the reranker, are seq2seq models. we describe here how we train and use them. ## language model in this section, we describe the architecture of the language models we used and how we trained them. ## experiments and results we use the original chinese subset of newstest2017 and newstest2018 as our validation set and new-stest2019 as our test set. the candidate translations for the reranker are generated by 8 ensemble models (6 from each). in our final submitted system, we tune the weights of component models and the hyperparameters of sentence segmentation models using a combination of our validation set and test set. for the candidate translations of the reranker, apart from the existing 48 proposals generated by 8 ensemble models, we include additional 48 proposals generated by 8 ensemble models which are finetuned with cwmt, newstest2017, newstest2018, and newstest2019. we also include translations generated by mcts decoding ( §b) in our nonprimary system. we find that adding a feature marking the length of source sentences longer than 60 words helps the reranker handle long sentences better. we therefore include this feature in addition to proposal probability, adversarial proposal probability, channel probability, language model probability, and length bonus (eqn. 3). our system achieves a 35.4 bleu score on new-stest2020. ## conclusion this paper describes the deepmind submission to the wmt2020 news chinese-english translation task. using the noisy channel model @xcite as our core document translation system, we optimized its component models using data augmentation, fine-tuning with in-domain data, mcts decoding ( §b), and knowledge distillation. we also addressed premature termination in long sentences by training specialized length expert models and segmenting long sentences into multiple shorter sentences ( §a). we have demonstrated the marginal contributions of these methods in our analysis and our final system comprising all these methods outperforms the transformer baseline by 9.9 bleu points on newstest2019. in the appendix, we additionally include the description of our specialized methods for handling long sequences and the mcts decoding algorithm. the candidate translations generated by mcts decoding are added to the candidate pool of the noisy channel reranker in our non-primary system.