Pre-training Methods for Neural Machine Translation (2021)

Abstract: This tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. Firstly, we will briefly introduce the background of NMT, pre-training methodology, and point out the main challenges when applying pre-training for NMT. Then we will focus on analysing the role of pre-training in enhancing the performance of NMT, how to design a better pre-training model for executing specific NMT tasks and how to better integrate the pre-trained model into NMT system. In each part, we will provide examples, discuss training techniques and analyse what is transferred when applying pre-training.

## tutorial introduction pre-training is a dominant paradigm in nature language processing (nlp) @xcite @xcite , computer vision (cv) @xcite and auto speech recognition (asr) @xcite @xcite . typically, the models are first pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware representation of the input, or initializing the parameters of the downstream model for fine-tuning. recently, the trend of self-supervised pre-training and task-specific fine-tuning finally fully hits neural machine translation (nmt) @xcite @xcite . despite its success, introducing a universal pretrained model to nmt is non-trivial and not necessarily yields promising results, especially for the resource-rich setup. unique challenges remain in several aspects. first, the objective of most pretraining methods are different from the downstream nmt tasks. for example, bert @xcite , a popular pre-trained model, is designed for language understanding with only a transformer encoder, while an nmt model usually consists of an encoder and a decoder to perform cross-lingual generation. this gap makes it not feasible enough to apply pre-training for nmt @xcite . besides, machine translation is naturally a multilingual problem, but general pre-training methods for nlp mainly focus on english corpus, such as bert and gpt. given the success of transfer learning in multi-lingual machine translation, it is very appealing to introduce multi-lingual pre-training for nmt @xcite . finally, speech translation has attracted much attention recently, while most pre-training methods are focused on text representation. how to leverage the pretraining methods to improve the speech translation becomes a new challenge. this tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. firstly, we will briefly introduce the background of nmt, pre-training methodology, and point out the main challenges when applying pre-training for nmt. then we will focus on analysing the role of pre-training in enhancing the performance of nmt, how to design a better pretraining model for executing specific nmt tasks and how to better integrate the pre-trained model into nmt system. in each part, we will provide examples, discuss training techniques and analyse what is transferred when applying pre-training. the first topic is the monolingual pre-training for nmt, which is one of the most well-studied field. monolingual text representations like elmo, gpt, mass and bert have superiorities, which significantly boost the performances of various natural language processing tasks @xcite @xcite . however, nmt has several distinct characteristics, such as the availability of large training data (10 million or larger) and the high capacity of baseline nmt models, which requires carefully design of pre-training. in this part, we will introduce different pre-training methods and analyse the best practice when applying them to different machine translation scenarios, such as unsupervised nmt, low-resource nmt and rich-source nmt @xcite . we will cover techniques to finetune the pre-trained models with various strategies, such as knowledge distillation and adapter @xcite . the next topic is multi-lingual pre-training for nmt. in this context, we aims at mitigating the english-centric bias and suggest that it is possible to build universal representation for different language to improve massive multi-lingual nmt. in this part, we will discuss the general representation of different languages and analyse how knowledge transfers across languages. these will allow a better design for multi-lingual pre-training, in particular for zero-shot transfer to non-english language pairs @xcite @xcite @xcite @xcite @xcite . the last technical part of this tutorial deals with the pre-training for speech nmt. in particular, we focus on leverage weakly supervised or unsupervised training data to improve speech translation. in this part, we will discuss the possibilities of building a general representations across speech and text. and shows how text or audio pre-training can guild the text generation of nmt @xcite @xcite @xcite @xcite @xcite . we conclude the tutorial by pointing out the best practice when applying pre-training for nmt. the topics cover various of pre-training methods for different nmt scenarios. after this tutorial, the audience will understand why pre-training for nmt is different from other tasks and how to make the most of pre-training for nmt. importantly, we will give deep analyze about how and why pre-training works in nmt, which will inspire future work on designing pre-training paradigm specific for nmt. ## tutorial breadth based on the representative set of papers listed in the selected bibliography, we anticipate that 70%-80% of the tutorial will cover other researchers' work, while the rest concerns the work where at least one of the presenters has been actively involved in. we will introduce several important work related to the monolingual, the multi-lingual and the multi-modal pre-training for nmt. ## diversity in the tutorial, some multilingual pre-training methods will scale to over 50 to 100 different languages. researchers working on the diverse language pairs might find this tutorial relevant and useful. ## prerequisites the tutorial is self-contained. we will address the background, the technical details and the examples. basic knowledge about neural networks are required, including word embeddings, attention, and encoder-decoder models. prior nlp courses and familarity with the machine translation task are preferred. it is recommended (and optional) that audience to read the following papers before the tutorial: ## target audience this tutorial will be suitable for researchers and practitioners interested in pre-training applications and multilingual nlp, especially for machine translation. to the best of our knowledge, this is the first tutorial that focuses on the pre-training methods and practice for nmt. ## technical requirements the tutorial will be online. internet connection with proper live video device is needed. our slides and video is open to public, available at https://lileicc.github.io/talks/ 2021-acl/ . 10 tutorial presenters mingxuan wang (bytedance ai lab) google scholar dr. mingxuan wang is a senior researcher at bytedance ai lab. he received his phd degree from the chinese academy of sciences institute of computing technology in 2017. his research focuses on natural language processing and machine translation. he has published over 20 papers in leading nlp/ai journals and conferences such as acl, aaai and emnlp. he has served in the program committee for acl/emnlp 2016-2020, aaai/ijcai 2018/2019, neurips 2020. he achieved outstanding results in various machine translation evaluation competitions, including the first place of chinese-to-english translation at at the wmt 2018, the third place of chinese-to-english translation at nist 2015, etc. together with dr. lei li, he is leading a team developing the volctrans machine translation system. he has given a tutorial about machine translation at ccmt 2017 and was an guest lecturer for 2016 machine translation for university of chinese academy of sciences (ucas). lei li (bytedance ai lab) https://lileicc.github.io/ dr. lei li is director of bytedance ai lab, leading the research and product development for nlp, robotics, and drug discovery. his research interests are machine translation, speech translation, text generation, and ai powered drug discovery. he received his b.s. from shanghai jiao tong university and ph.d. from carnegie mellon university, respectively. his dissertation work on fast algorithms for mining co-evolving time series was awarded acm kdd best dissertation (runner up). his recent work on ai writer xiaomingbot received 2nd-class award of wu wen-tsün ai prize in 2017. he is a recipient of ccf distinguished speaker in 2017, and ccf young elite award in 2019. his team won first places for five language translation directions in wmt 2020 and the best in corpus filtering challenge. before bytedance, he worked at eecs department of uc berkeley and baidu's institute of deep learning in silicon valley. he has served organizers and area chair/senior pc for multiple conferences including kdd, emnlp, neurips, aaai, ijcai, and cikm. he has published over 100 technical papers in ml, nlp and data mining and holds more than 10 patents. he has started and is developing bytedance's machine translation system, volctrans and many of his algorithms have been deployed. he has delivered four tutorials at emnlp 2019, nlpcc 2019, nlpcc 2016, and kdd 2010. he was an lecturer for 2014 probabilistic programming for advancing machine learning summer school at portland, usa. ## other information prior related tutorials neural machine translation, presented by thang luong, kyunghyun cho, and christopher manning at acl 2016. this tutorial is related but different from acl 2016 nmt tutorial. it focuses on pre-training methods for both bilingual, multi-lingual, and multi-modal neural machine translation. unsupervised cross-lingual representation learning, presented by sebastian ruder, anders søgaard, and ivan vulić at acl 2019. this tutorial is related in concerning multi-lingual nlp. however, their tutorial was on representation learning, while our tutorial is on neural machine translation.