The Fujitsu DMATH Submissions for WMT 21 News Translation and Biomedical Translation Tasks (2021)

Abstract: This paper describes the Fujitsu DMATH systems used for WMT 2021 News Translation and Biomedical Translation tasks. We focused on low-resource pairs, using a simple system. We conducted experiments on English-Hausa, Xhosa-Zulu and English-Basque, and submitted the results for Xhosa→Zulu in the News Translation Task, and English→Basque in the Biomedical Translation Task, abstract and terminology translation subtasks. Our system combines BPE dropout, sub-subword features and back-translation with a Transformer (base) model, achieving good results on the evaluation sets.

## introduction wmt has been exploring the state of the art in mt for many years, and, particularly in recent editions, the participants have shown impressive results. however, often times, these results require very heavy or complex systems, trained on dozens of gpus. participants compete for a margin that places them above the rest, combining multiple methods from the latest research. in recent years, different variants of the transformer @xcite architecture have been popular for nmt, so can be seen when inspecting the submissions to previous editions of wmt. in our systems, we use the transformer base configuration, the smaller one. our implementation is based on sockeye 2 @xcite . we combine several techniques or strategies for low-resource pairs. these techniques are described in section 2. we conducted a few experiments on language pairs xhosa-zulu and english-hausa, from the news transltion task, and on english-basque, from the biomedical translation task. the results of our experiments are shown in section 3. ## techniques this section describes the strategies used for our nmt models. the first two, bpe dropout and subsubword features, were used in all the subtasks, while the last one was only used for the biomedical translation subtasks. ## experiments we conducted experiments on @xmath0, @xmath1, @xmath2, @xmath3. notice that the wmt21 biomedical translation task for english-basque was only in the @xmath4, and not @xmath5. word ratios can hint about the similarity or dissimilarity of the languages. xhosa and zulu are related languages, and that is why they show a ra- we used 4,000 warmup steps schedule as described in @xcite with an initial learning rate of 2.0 and evaluated the development cost every 2,000 updates. the model was reloaded from the best checkpoint when the development cost did not improve, and training stopped after 3 consecutive stallings. hyperparameter value vocabulary size 32,000 subwords bpe dropout p 0.1 batch size 4,096 (×2 gpus) warmup steps 4,000 learning rate 2.0 encoder layers 6 decoder layers 6 attention heads 8 transformer size 512 hidden layer size 2,048 dropout 0.1 label smoothing ϵ 0.1 fte layers † 3 fte size † 3,072 for the news translation task participants need agree to contribute to the manual evaluation about eight hours of work, per system submission. in consideration of this workload, we decided to sub-mit only the @xmath6. the results show that the sub-subword features (+sswf) improve the results of the corresponding -sswf models under low-resource settings. in the case of @xmath7, the +sswf system did not achieve better bleu scores than the corresponding -sswf system, but achieved better chrf2. despite its noisy nature, we decided to use the extended common crawl hausa corpus. the results show that the data, although noisy, was effective in improving the performance. the @xmath8. it is possible that the cause for this was the domain mismatch of the monolingual data, that was not exclusively from scientific papers' abstracts. all models were trained on two nvidia tesla p100 gpus. the xhosa-zulu models are trained in about 2.5 hours, and the english-hausa models are trained in about 10 hours. the same models were used for abstract translation and terminology translation. manual examination of the produced transations hinted better performance for the the model trained with english-spanish terminology. in consideration of the results, we decided to submit two systems to the abstract translation and terminology translation subtasks. one of the systems incorporated the mespen terminology, and the other one did not. both systems did not use backtranslated data. ## conclusions we built and submitted three lightweight systems that used sub-subword features to build the embeddings. we evaluated the approach with different configurations and the results showed the adequacy of the approach. the relatively small models could possibly use larger hyperparameters and other techniques to achieve better results, but we think the current results can show the strenght of the techniques that were applied.