Small Model and In-Domain Data Are All You Need (2021)

Abstract: I participated in the WMT shared news translation task and focus on one high resource language pair: English and Chinese (two directions, Chinese to English and English to Chinese). The submitted systems (ZengHuiMT) focus on data cleaning, data selection, back translation and model ensemble. The techniques I used for data filtering and selection include filtering by rules, language model and word alignment. I used a base translation model trained on initial corpus to obtain the target versions of the WMT21 test sets, then I used language models to find out the monolingual data that is most similar to the target version of test set, such monolingual data was then used to do back translation. On the test set, my best submitted systems achieve 35.9 and 32.2 BLEU for English to Chinese and Chinese to English directions respectively, which are quite high for a small model.

## introduction i participated in the wmt shared news translation task and focus on the english and chinese language pair. this language pair is challenging due to the plentiful in-domain bitext training data and abundant monolingual data. high resource means fierce competition, many high-tech companies and universities chose this language pair also. my neural machine translation system is developed using base transformer @xcite architecture and the toolkit i used is thumt @xcite . rules and word aligning model are used to clean parallel data. language model is used to clean monolingual 1 http://mteval.cipsc.org.cn:81/agreement/description data. i use a base transformer @xcite architecture since i have only one gpu. the following techniques are used on model training: a. increase the number of encoder layers to 12 to further improve the encoder's representation capability; b. back translation @xcite are applied to fully utilize the monolingual corpus. c. shared vocabulary is used for better performance. d. four different models using diversified data are trained for ensemble decoding. ## data filtering and selection the parallel data is mainly from ccmt corpus 1 , and the monolingual data is collected from the internet. i did not use any other datasets since i think they are not highly related to this news translation task. to evaluate my model's performance, i merged the test set from wmt2017 to wmt2020 to build a big development set. ## system description this section illustrate how i train the model step by step. ## conclusion this paper describes hui zeng's translation systems (zenghuimt) for the wmt2021 news translation shared task. the potential of small model plus in-domain data is explored. i am pleased to argue that, with high quality in-domain data, small model could achieve bleu scores comparable to that of huge models.