  Recent studies have demonstrated a perceivable improvement on the performance
of neural machine translation by applying cross-lingual language model
pretraining (Lample and Conneau, 2019), especially the Translation Language
Modeling (TLM). To alleviate the need for expensive parallel corpora by TLM, in
this work, we incorporate the translation information from dictionaries into
the pretraining process and propose a novel Bilingual Dictionary-based Language
Model (BDLM). We evaluate our BDLM in Chinese, English, and Romanian. For
Chinese-English, we obtained a 55.0 BLEU on WMT-News19 (Tiedemann, 2012) and a
24.3 BLEU on WMT20 news-commentary, outperforming the Vanilla Transformer
(Vaswani et al., 2017) by more than 8.4 BLEU and 2.3 BLEU, respectively.
According to our results, the BDLM also has advantages on convergence speed and
predicting rare words. The increase in BLEU for WMT16 Romanian-English also
shows its effectiveness in low-resources language translation.
