NLPRL System for Very Low Resource Supervised Machine Translation (2020)

Abstract: This paper describes the results of the system that we used for the WMT20 very low resource (VLR) supervised MT shared task. For our experiments, we use a byte-level version of BPE, which requires a base vocabulary of size 256 only. BPE based models are a kind of sub-word models. Such models try to address the Out of Vocabulary (OOV) word problem by performing word segmentation so that segments correspond to morphological units. They are also reported to work across different languages, especially similar languages due to their sub-word nature. Based on BLEU cased score, our NLPRL systems ranked ninth for HSB to GER and tenth in GER to HSB translation scenario.

## introduction we report the results for our system that was used for our participation in the wmt20 shared task @xcite on very low resource machine translation (mt). the mt systems were built for the language pair upper sorbian (hsb) and german (ger) in both translation directions. the sorbian languages are the west slavic branch of the indo-european languages, which have further categorized into two closely related languages, upper sorbian and lower sorbian. the categories of this language are recognized as a different and distinct language in the european charter for regional or minority languages @xcite . upper sorbian is a minority language of germany that is spoken by 10, 000 to 15, 000 speakers @xcite , although this number is continually declining (dołowy-rybińska, 2018). to counter this, attempts are being made to increase the number of sorbian speakers through bilingual educational scenarios and mt foot_0 . low resource mt was being attempted even before neural machine translation (nmt) became the state-of-the-art. several methods are used to improve the accuracy and quality of the lowresource smt systems by using comparable corpora @xcite , pivot language (english or non-english) technique @xcite , and using related resource-rich language @xcite . we use a byte-level version of byte pair encoding based model with a transformer for our experiments. the main motivation was to try out this model for the shared task and see how it works under a shared task setting. ## background nmt is an end-to-end learning system @xcite , based on the data-driven approach of machine translation, that requires a massive amount of parallel data for training. to overcome the lack of such data, several techniques have been tried out which are based on semi-supervised learning ( zheng et al. (2019) ), unsupervised learning @xcite ), data augmentation ( siddhant et al. (2020) ), transfer learning (aji (2020)), meta-learning ( li et al. (2020) ), pivot-based ( kim et al. (2019) ), and multilingual machine translation ( dabre et al. (2020 ## system description the standard transformer architecture proposed by @xcite is used for this experiment. this architecture is able to handle long-term dependencies among input tokens, output tokens and between input-output by multi-head attention mechanism. our method based on the model architecture of @xcite , which had used the byte-level bpe (bbpe). the bbpe encoding is deployed on the byte pair encoding (bpe) @xcite , which is a subword algorithm to find a way to represent the given entire text dataset with a small number of tokens. bpe tries to find a balance between characterand word-level hybrid representations, enabling the encoding of any rare words in the vocabulary with appropriate subword tokens without introducing any "unknown" tokens. these segmented byte sequences are encoded into variable-length tokens, i.e., n-grams, which leads to the generation of the bpe vocabulary with byte n-grams. before being fed to the transformer model, the learned bbpe passes through bidirectional gru, which enables to retain contextualization between byte representation of bpe. ## experimental setup we use the fairseq foot_1 @xcite library to train the transformer with the same learning rate as in the original paper. ## results and analysis the bbpe based transformer model was evaluated on the blind test set at five different metrics provided by the task organizer, namely bleu @xcite , bleu-cased, ter @xcite , beer2.0 @xcite , and character @xcite . the obtained metrics score for each pair to each experiment is specified in we have report the results for a transformer-based mt system for the pair of hsb↔ger in very low resource settings.