  Previous works mainly focus on improving cross-lingual transfer for NLU tasks
with multilingual pretrained encoder (MPE), or improving the translation
performance on NMT task with BERT. However, how to improve the cross-lingual
transfer of NMT model with multilingual pretrained encoder is under-explored.
In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In
this task, the NMT model is trained with one parallel dataset and an
off-the-shelf MPE, then is directly tested on zero-shot language pairs. We
propose SixT, a simple yet effective model for this task. The SixT model
leverages the MPE with a two-stage training schedule and gets further
improvement with a position disentangled encoder and a capacity-enhanced
decoder. The extensive experiments prove that SixT significantly improves the
translation quality of the unseen languages. With much less computation cost
and training data, our model achieves better performance on many-to-English
testsets than CRISS and m2m-100, two strong multilingual NMT baselines.
