Unsupervised and Very-Low Resource Supervised Translation on G erman and Sorbian Variant Languages (2022)

Abstract: This paper presents the work of team PICT-NLP for the shared task on unsupervised and very low-resource supervised machine translation, organized by the Workshop on Machine Translation, a workshop in collocation with the Conference on Empirical Methods in Natural Language Processing (EMNLP 2022). The paper delineates the approaches we implemented for supervised and unsupervised translation between the following 6 language pairs: German-Lower Sorbian (de-dsb), Lower Sorbian-German (dsb-de), Lower Sorbian-Upper Sorbian (dsb-hsb), Upper Sorbian-Lower Sorbian (hsb-dsb), German-Upper Sorbian (de-hsb), and Upper Sorbian-German (hsb-de). For supervised learning, we implemented the transformer architecture from scratch using the Fairseq library. Whereas for unsupervised learning, we implemented Facebook’s XLM masked language modeling approach. We discuss the training details for the models we used, and the results obtained from our approaches. We used the BLEU and chrF metrics for evaluating the accuracies of the generated translations on our systems.

## introduction neural machine translation has witnessed significant progress in the case of highly spoken languages such as english @xcite , mandarin @xcite , french (emezue and dossou, 2020), etc. however, in many cases, it becomes challenging to develop a robust bilingual machine translation system, especially with limited resources @xcite . there are big-tech companies such as google 1 and bing 2 , which have taken initiatives to build translation systems for * equal contribution † equal contribution ‡ equal contribution 1 https://translate.google.com/ 2 https://www.bing.com/translator multiple languages. still, languages that are lowresource in nature, such as the sorbian family of languages @xcite , have gotten lesser attention in terms of research. the paper focuses on the development of machine translation systems between pairs of languages from german, lower sorbian, and upper sorbian, using both supervised and unsupervised approaches. in the indo-european language family, german (deutsch) belongs to the western germanic branch @xcite . approximately 95 million people speak it natively; 28 million speak it as a second language in more than 40 countries. due to a phonetic mutation called high german consonant shift @xcite , german moved away from other germanic languages. this shift in german consonants occurred between the 3rd and 5th centuries, and probably ended in the 9th century ad. lower sorbian (dolnoserbska rěc) and upper sorbian (hornjoserbska rěč) are western slavonic languages spoken in the region of lower and upper lusatia in the southeast of germany respectively. they are closely related to other west slavonic languages, including polish, czech, slovak, and kashubian. there are seven recognized autochthonous minorities and regional languages in germany, including danish, saterfrisian, north frisian, romanes, and lower german. we aimed to carry out research in neural machine translation between german, which is highresource in nature, and lower and upper sorbian, which are low-resource languages. we implement supervised and unsupervised methods for the same. for the supervised approach, we trained transformer @xcite models from scratch using the bilingual data provided by wmt in the 2022 workshop edition. we used the fairseq foot_0 @xcite library for the same, which is a sequence-to-sequence learning toolkit for neural supervised de-dsb dsb-de dsb-hsb hsb-dsb de-hsb hsb-de parallel 40,194 40,194 62,565 62,565 70,000 70,000 for the unsupervised approach, implemented facebook xlm's foot_1 masked language model (crosslingual language model) for unsupervised learning @xcite . training data used for the same was monolingual data provided by the organizers and the opus project foot_2 . we preprocessed the data, and also applied byte-pair encoding (bpe) @xcite to the input and target data. we made use of the fastbpe foot_3 library for the same. finally, we applied xlm preprocessing on the data before training. we experimented our approaches on six language pairs between german, lower sorbian and upper sorbian. we have used the bleu @xcite and chrf @xcite evaluation metrics for computing accuracy, which have been discussed in this paper. ## dataset description we used the data provided by the wmt22 organizers, and from the opus project, recommended by the organizers. the statistics of the training data for both supervised and unsupervised approaches is given in for unsupervised learning, we used the monolingual data for lower sorbian provided by the organizers. for upper sorbian, we used the monolingual data provided by the witaj sprachzentrum foot_4 . whereas for german, we used the monolingual data provided by the opus project. the quantitative statistics of these datasets is given in the blind test data provided by the organizers contained 1000 sentences each for translations between lower sorbian and german, 1000 sentences each for translations between lower sorbian and upper sorbian, and 1621 sentences each for translations between upper sorbian and german. we submitted the inferences on the blind test data to the shared task leaderboard. ## data preparation the data preprocessing step was crucial in determining the accuracies of our translations. the goal was not to waste resources (compute power, time) in processing things that don't add much value to extracting the semantics and understanding the text. @xcite for supervised learning, we preprocessed the source and target text using fairseq-preprocess foot_5 , an inbuilt preprocessing script provided by fairseq. we set the number of parallel workers for preprocessing the text as 20, so as to achieve faster preprocessing. normalization @xcite and pre-tokenization of the text is done before passing the to fairseq-preprocess. we used sacremoses 9 tokenizer, which helps us to tokenize and normalize text according to our needs. fairseq-preprocess binarizes the training data and builds vocabularies from the text of that particular language. for unsupervised learning, we applied some additional preprocessing, which consisted of using xlm-moses tokenizer. the xlm-moses tokenizer performs the following steps: removing unicode punctuations, normalizing punctuations (punctuations will be removed from the utterances during training) and removing any non-printing characters. we also applied byte-pair encoding @xcite to our data, where we use the inbuilt script provided by fastbpe foot_7 . byte-pair encoding algorithm computes the unique set of words used in the corpus (after the normalization and pretokenization steps are completed), and then builds the vocabulary by taking all the symbols used to write those words. byte-pair encoding algorithm application includes learning the bpe codes from the training dataset, then applying the same on the training, validation and test datasets. also, we get the training vocabulary once we have obtained the codes after training. finally, we apply xlm preprocessing provided by facebook xlm, to get the final preprocessed data. ## results for the results, please refer to ## related work machine translation has been a pivotal field of research in the natural language processing domain. with rule-based and statistical machine translation methods proposed in the past decades, neural machine translation has surpassed these conventional methods by achieving state-of-the-art accuracies with each year. in 2014, bahdanau @xcite proposed the base paper for neural machine translation. according to the paper, the encoder part of the model encodes the input sentence into a fixed length vector, from which the decoder generates the translation. the encoder and decoder parts could be neural architectures such as a simple rnn, lstm @xcite , bidirectional rnn @xcite , gru @xcite , etc. with the introduction of transformers @xcite and self-attention in training neural networks, nmt research got a substantial boost. german, being pretty high resource in nature; there has been significant work carried out in german in nmt. the workshop on machine translation (wmt) has a significant contribution to the same. minh-thang luong @xcite demonstrate two seperate attention mechanisms (global and local attention) for bidirectional translations between english and german, gaining an increase of 5.0 in the bleu score over non-attention based techniques. macketanz @xcite present the result of applying a fine-grained test suite on the outputs of 36 state-of-the-art machine translation systems between english and german, which were submitted to the sixth conference on machine translation. xu @xcite proposed bibert, a bilingual bert model which helped in achieving state-of-the-art translation performance compared to other published papers till date, and that too without implementing backtranslation @xcite . the paper also proposes a stoicastic layer selection method which helps in improving translation performance. sorbian family of languages have started receiving attention with regards to nmt research in the past few years. li and team @xcite worked on supervised machine translation for a few language pairs, which included german-upper sorbian translations. they experimented with document-enhanced nmt, xlm pretrained language model enhanced nmt, etc. their primary submissions won the first place in the german to upper sorbian translation directions. knowles and team @xcite worked on implementing ensemble learning in transformer models for german-upper sorbian, built using bpe-dropout, lexical modifications and backtranslation. pertaining to unsupervised learning: chronopoulou @xcite propose the lmu munich system for the wmt 2020 unsupervised machine translation task, which involves using a pretrained monolingual model and finetuning it on both german and upper sorbian. finally, the system uses backtranslation, and uses the pseudo-parallel data obtained to finetune the model further. finally, the paper ensembles the best best-performing systems and give state-of-the-art scores on unsupervised translations between german and upper sorbian. edman @xcite implement transformer encoder-decoder architectures for unsupervised nmt from german to lower sorbian. the system has three modifications from the conventional methodology-training followes a bilingual approach, instead of a multilingual system approach. secondly, a novel method is introduced for building the vocabulary of an unseen language. finally, experimentation is done with the order of implementation of online and offline backtranslation. the paper received first place in the unsupervised machine translation task for wmt 2021. ## conclusion thus, we have implemented supervised and unsupervised neural machine translation approaches for translation between language pairs consisting of german(de), lower sorbian(dsb), and upper sorbian(hsb). we utilized different architectures for implementing the same. our future plans include training these models with much larger corpora on computationally-efficient machines to obtain better evaluation metric scores and use high-end gpus for practical training. we plan to use better preprocessing techniques and linguistic methods to improve the usefulness of the final training data to be fed to the models. we plan to implement backtranslation to improve current translation accuracy, have longer pre-training, and implement other pretrained models such as mbert and xlm.