NRC - CNRC Systems for U pper S orbian- G erman and L ower S orbian- G erman Machine Translation 2021 (2021)

Abstract: We describe our neural machine translation systems for the 2021 shared task on Unsupervised and Very Low Resource Supervised MT, translating between Upper Sorbian and German (low-resource) and between Lower Sorbian and German (unsupervised). The systems incorporated data filtering, backtranslation, BPE-dropout, ensembling, and transfer learning from high(er)-resource languages. As measured by automatic metrics, our systems showed strong performance, consistently placing first or tied for first across most metrics and translation directions.

## introduction this work describes our machine translation (mt) systems for translating between upper sorbian-german and lower sorbian-german (all translation directions). we focused primarily on the supervised task of upper sorbian-german, and then applied those systems to the task of building simple lower sorbian-german systems. @xcite upper sorbian and lower sorbian are slavic minority languages spoken in eastern germany, alongside german. the shared task data was provided to the organizers through collaborations with the sorbian institute 2 and the witaj language centre, 3 as described in fraser (2020), to which we direct interested readers for additional information on the languages and data. following the 2020 shared task, the witaj language centre released a publicly-available sorbian-german mt system sotra (witaj language centre, 2021) based on moses (koehn et al., 2007) and opennmt (klein et al., 2017). @xcite we provide an overview of the data, preprocessing, and model architectures in sections 2, 3, and 4. we then discuss baselines, systems, experiments in monolingual filtering, and backtranslation (all focused on upper sorbian-german) in sections 5, 6, 7, and 8. in section 9, we discuss how we applied and finetuned our existing upper sorbian mt systems for the task of translating lower sorbian. section 10 discusses additional experiments with negative results. finally, sections 11 and 12 summarize the final systems and our conclusions. ## data we used all provided parallel german-upper sorbian data and all monolingual upper sorbian data (after filtering), along with german-czech parallel data from open subtitles (lison and tiedemann, 2016), foot_0 dgt (tiedemann, 2012; steinberger et al., 2012), jw300 (agić and vulić, 2019), europarl v10 (koehn, 2005), news-commentary v15, and wmt-news. @xcite we also used the monolingual upper sorbian web, witaj and sorbian institute datasets as well as the lower sorbian monolingual data (the latter for lower sorbian tasks only). @xcite we used the provided devel sets for development, and the devel_test systems for measuring progress and choosing which systems to submit. ## preprocessing and postprocessing as preprocessing, we first clean all of the available training data (but not development or test data) using clean-utf8-text.pl with the -no-phrase-sep flag from portagetextprocessing. @xcite for parallel training data, we use clean-corpus-n.perl from moses (koehn et al., 2007) with ratio 15, and for monolingual data we remove empty lines. we normalize punctuation with moses's normalize-punctuation.perl and remove the non-breaking space \xa0. we perform additional sentence splitting to improve tokenization, @xcite then tokenize with moses's tokenizer.perl -a -l @xmath0lng is cs, de, or hsb), then re-merge the sentences that were split into single lines. for all german-czech parallel data and all monolingual german or czech data, we removed any lines that contained characters that had not been observed in de-hsb training data, wmt-news, or europarl. this helps clean data of unusual encoding issues, as well as removing text that is clearly in other languages (i.e., written in other scripts). we build bpe vocabularies of size 10k, 15k, 20k, and 25k merges using subword-nmt @xcite . we also add all moses and sockeye special tags (ampersand, <unk>, etc.) and a number of additional reserved tags (for backtranslation, languages, etc.) to a glossary file used for applying bpe, which prevents them from being segmented. for building the bpe models, we used all hsb-de data, the sorbian institute and witaj monolingual hsb data, cs-de data, and news-commentary (de) data; the hsb data was upscaled twice (see in standard postprocessing, we de-bpe and detokenize (using the moses detokenizer.perl -a -l $lng). ## models we built transformer models (vaswani et al., 2017) using sockeye (hieber et al., 2018) version 2.3.14 and cuda-10.1. we used the default value of 6 encoder/decoder layers, 8 attention heads, the adam (kingma and ba, 2015) optimizer, label smoothing of 0.1, a cross-entropy-without-softmax-output loss, and a model size of 512 units with a ffn size of 2048. we performed early stopping after 32 checkpoints without improvement. we chose custom checkpoint intervals of 4000 updates when the train corpus was deemed big enough and 500 updates when the train corpus was small. we optimized for bleu (papineni et al., 2002) @xcite and used the whole validation set during validation. the batch size was set to 8192 tokens, and the maximum sequence length for both source and target was set to 200 tokens. we used weight tying and vocabulary sharing, but we set gradient clipping to absolute and kept the initial learning rate of 0.0002. we used a beam size of 5 in all submit- @xcite all bleu scores were computed using sacrebleu (post, 2018) with the signature bleu+case.mixed+numrefs.1+smooth.exp+ tok.13a+version.1.4.14. the chrf scores (popović, 2015) were generated in the submission interface. ted systems. @xcite when systems deviate from these (i.e., different learning rates or label smoothing), we make note of it in our descriptions. ## baselines we build several types of baselines against which to measure our improvements. they are shown in ## systems here we describe the general types of systems that we have built, including parent de-cs systems, multilingual systems, and the child (and grandchild) systems we built on top of those. ## monolingual data filtering given the tight coupling between the domain of the development/development-test data and the training data, the large quantity of monolingual data available for backtranslation from german, and inspired by the filtering used in the high-performing 2020 submission by scherrer et al. (2020) we examined whether we should subsample data for backtranslation. @xcite we used the 2020 final systems described in knowles et al. (2020) to backtranslate all available news 2019 de. this enabled us to train child systems with a random sample of 1.5 million lines of text, the full available backtranslated data, and several approaches to sampling the data. we first describe hsb-de experiments with the fixed data size of 1.5 million lines of backtranslated de monolingual data. we use a random sample as a baseline. we compare to it two approaches to using pretrained sentence-transformer embeddings @xcite (reimers and gurevych, 2019) and cosine similarity for domain filtering: ranking sentences in the monolingual data based on their similarity to the average embedding of the full de side of @xcite as described in section 7, moore-lewis filter. @xcite data subsampling or filtering or of one sort or another was also used by several other submissions in 2020, including: dutta et al. (2020), edman et al. (2020), and knowles et al. (2020). @xcite from https://github.com/ukplab/ sentence-transformers , with model paraphrasexlm-r-multilingual-v1, a multilingual version of paraphrasedistilroberta-base-v1, trained on parallel data for 50+ languages (reimers and gurevych, 2020). ## inconclusive and negative results we now discuss negative results, i.e., experiments that we performed that were unsuccessful. all of these were performed on upper sorbian-german. ## final systems according to preliminary automatic metric results from the shared task organizers, our systems performed quite well. the metrics considered were bleu, chrf, and -in the case of translation into german -bert score. each translation direction saw five systems submitted, with the exception of dsb-de, which only had four. our hsb-de had the best bert score (0.981), the second-best bleu score (67.3, 0.4 bleu behind noahnmt), and the best chrf score; it was significantly better than all four other systems in terms of bert score, while in terms of bleu and chrf it was better than three other systems (tying with noahnmt). our dsb-de system had the best bleu and chrf scores and was tied for the best bert score (0.953) with cl_rug; in terms of bleu and chrf it was significantly better than one other system (alongside cl_rug and lmu) and in terms of bert score it was significantly better than two other systems (alongside cl_rug). both of our systems translating out of german had the highest bleu and chrf scores. our de-hsb system was, alongside noahnmt, significantly better than three other systems in both automatic metrics. our de-dsb system was, alongside lmu, significantly better than three other systems in both automatic metrics. ## conclusions as with last year's task, we found that our best systems consisted of ensembles, with more diverse ensembles performing better than less diverse ones. the very high automatic metric scores along with our experiments in backtranslation led us to examine the state of memorization of the training data, which we found to be quite high. we also found that the close relationship between upper sorbian and lower sorbian enabled us to bootstrap seemingly strong lower sorbian systems through iterative backtranslation. we believe that the true test of these systems will be through human evaluation, as well as an analysis of how well they perform in a real-life setting (i.e., with more out-of-domain test data), as the current set seems potentially quite constrained in domain.