MUNI - NLP Systems for L ower S orbian- G erman and L ower S orbian- U pper S orbian Machine Translation @ WMT 22 (2022)

Abstract: We describe our neural machine translation systems for the WMT22 shared task on unsupervised MT and very low resource supervised MT. We submit supervised NMT systems for Lower Sorbian-German and Lower Sorbian-Upper Sorbian translation in both directions. By using a novel tokenization algorithm, data augmentation techniques, such as Data Diversification (DD), and parameter optimization we improve on our baselines by 10.5-10.77 BLEU for Lower Sorbian-German and by 1.52-1.88 BLEU for Lower Sorbian-Upper Sorbian.

## data we experiment with lower sorbian-german and lower sorbian-upper sorbian translation, using only the parallel data provided for each pair. the parallel data for the dsb-de and the dsb-hsb pairs consist of ∼40k and ∼62k sentences respectively. we use only these data, as the approach we decided to follow does not need additional data. after applying this method, our final corpus size for training is ∼360k sentences for dsb-de, and ∼560k for dsb-hsb. ## methodology in this section, we first present briefly our novel tokenizer, high frequency tokenizer, or hft. then, we describe the architecture of our models and how we trained them. @xcite showed that a meaningful subword tokens vocabulary is crucial for achieving good performance in low-resource nmt. while they experiment with bpe, we employ our novel tokenization algorithm, high frequency tokenizer, or hft. this word segmentation methods aims to provide more meaningful subword dictionaries by obtaining more frequent, and thus @xmath0. given the importance of tokenization for low-resource nmt, we argue that is an important point to consider. ## final systems for all pairs and directions we worked on, our best systems was t-hft-dd, a transformer trained on a single nvidia a40 foot_4 for 50 epochs on dd hft-tokenized data, with a vocabulary size of 4k, feed-forward dimension of 2048, dropout 0.3, and 10240 for each batch. 6 our hypotheses on why hft leads to improvements on these datasets are the following. on top of providing more frequent and better defined token for the model to learn, it also explicitly marks both beginning and end of the words during pretokenization. this could be relevant for morphologically complex languages, such as the ones in this task, since it provides more information to the model on possible prefixes and suffixes. contrast this with the fact that subword-nmt bpe only explicitly mark continuation with the @@ marker. this kind of tokenization thus makes no distinction between full words and word endings. moreover, it seems to struggle with capitalized words and punctuation, which can also be informative, if handled optimally. hft's pretokenization seems to help with this issue. investigating these topics will be further addressed by future work. ## conclusions this paper described our submission for the wmt22 shared task on unsupervised mt and very low resource supervised mt. we presented systems for lower sorbian-german and lower sorbian-upper sorbian translation in both direction under supervised training conditions. to train our best systems we employed a novel tokenization algorithm, hft, to obtain more meaningful subword vocabularies, contrasted to a bpe baseline; and data diversification @xcite to augment the training data using only the parallel dataset provided for each language pair. variably on a a40, a100 or a tesla t4. training times did not exceed 12 hours for both t-[tok]-dd systems. during our experiments we confirmed that optimizing not only the transformer's hyper parameters, but also the subword vocabulary quality and size, are crucial steps for low-resource nmt. choosing the appropriate vocabulary size for the dataset, could lead to significant improvements in bleu score even with a small amount of parallel data. these, however, are still open and complex problems, since previously proposed settings or, even more so default ones, did not always provide the best results.