NRC Systems for the 2020 I nuktitut- E nglish News Translation Task (2020)

Abstract: We describe the National Research Council of Canada (NRC) submissions for the 2020 Inuktitut-English shared task on news translation at the Fifth Conference on Machine Translation (WMT20). Our submissions consist of ensembled domain-specific finetuned transformer models, trained using the Nunavut Hansard and news data and, in the case of Inuktitut-English, backtranslated news and parliamentary data. In this work we explore challenges related to the relatively small amount of parallel data, morphological complexity, and domain shifts.

## introduction we present the national research council of canada (nrc) inuktitut-english 1 machine translation (mt) systems in both translation directions for the 2020 wmt shared task on news translation. inuktitut is part of the dialect continuum of inuit languages, the languages spoken by inuit, an indigenous people whose homeland stretches across the arctic. included in this continuum are indigenous languages spoken in northern canada, including but not limited to the territory of nunavut. the term inuktut is used by the government of nunavut (2020) to describe inuit languages spoken in nunavut, such as inuktitut and inuinnaqtun. the majority of the inuit language text provided for the shared task comes from ᓄᓇᕗᒥ ᒪᓕᒐᓕᐅᕐᕕᐊ (nunavut maligaliurvia; legislative assembly of nunavut) through the nunavut hansard, the published proceedings of the legislative assembly of nunavut. the nunavut hansard is released publicly by the government of nunavut in inuktitut and english (also an official language of nunavut), and with their generous assistance was recently @xcite abbreviated iu and en using iso 639-2 codes. ## related work and motivation initial experiments on building neural machine translation (nmt) systems for inuktitut-english using the most recent nunavut hansard corpus are reported in joanis et al. (2020). earlier work includes micher (2018) and schwartz et al. (2020), and, predating the recent wave of nmt, martin et al. (2003), martin et al. (2005), and langlais et al. (2005). there has also been work on morphological analysis of inuktitut, including farley @xcite though we note that inuktitut, inuinnaqtun, english, and french may all be spoken in the house, we use the term inuktitut in describing our mt systems for two main reasons: 1) the official website describes the nunavut hansard as being published "in both inuktitut and english" (legislative assembly of nunavut, 2020) and 2) because we wish to make clear the limitations of our work; there is no reason to expect that the systems built using the data provided for wmt will perform well across various inuit languages and dialects (or even across a wider range of domains). @xcite computed using sacrebleu version 1.3.6 (post, 2018) with mteval-v13a tokenization: bleu+case.mixed+numrefs.1+smooth.exp+tok.13a. ## data while the parallel text size for this language pair is quite small compared to high-resource language pairs in the news translation task, inuktitut is one of the few indigenous languages in canada (or possibly the only) for which there exists enough parallel text (with any other language) to train robust statistical or nmt systems outside of the strictly low-resource paradigm (littell et al., 2018). thus we expect that it should be helpful to incorporate available monolingual data. we trained our baseline models using the full 1.3 million line nunavut hansard 3.0 (nh) parallel corpus. for iu-en, we also used a random subselection of 1.3m sentences of english europarl v10 (koehn, 2005; tiedemann, 2012) and 1.3m sentences of english 2019 news data 8 backtranslated into inuktitut (section 5.4). we did not use wiki titles or common crawl inuktitut data. @xcite we incorporated the news portion of the development data in training our models to alleviate the domain mismatch issue (section 5.1). ## preprocessing and postprocessing we first applied an internal script to convert control characters to spaces as well as normalizing spaces and hyphens; this was effectively a no-op for the nunavut hansard parallel corpus, but removed some problematic characters in the monolingual training data. parallel training corpora were then cleaned with the moses clean-corpus-n.perl script (koehn et al., 2007), using a sentence length ratio of 15:1 and minimum and maximum lengths of 1 and 200, respectively. for monolingual training data, the second cleaning step consisted of removing empty lines. for inuktitut, we used the normalize-iu-spelling.pl script provided by the organizers. we then performed punctuation normalization. this included specific en and iu normalization scripts, to more accurately capture and retain information about directional quotation marks, different types of dashes, and apostrophes, normalizing to the most common form. for inuktitut, this included treating word-internal apostrophes as u+02bc modifier letter apostrophe. @xcite appendix c provides a detailed description. after this preliminary normalization, we applied the moses normalize-punctuation.perl script, with the language set to en (or backing off to en, as there are currently no inuktitut-specific rules implemented). having noted that some of the lines in the training data contained more than one sentence (which results in unintended tokenization behavior), we next performed sentence splitting using the portage sentence splitter (larkin et al., 2010) on each side of the training data before feeding it to the moses tokenizer (using aggressive hyphen splitting). sentences that had been split were then re-merged following tokenization. we trained joint byte-pair encoding (bpe; sennrich et al., 2016c) models on the full nunavut hansard parallel training data using subword-nmt, then extracted english and inuktitut vocabularies separately. @xcite using a joint bpe model improves performance on roman text in inuktitut output (section 5.2 and appendix b). as postprocessing, we de-bpe the data, run the moses detokenizer, and then convert the placeholder tokens from our normalization scripts to their corresponding symbols (dashes, apostrophes, quotation marks, etc.). 12 ## experiments all models were typical transformers (vaswani et al., 2017) with 6 layers, 8 attention heads, network size of 512 units, and feedforward size of 2048 units, built using sockeye (hieber et al., 2018) version 1.18.115. we have changed the default gradient clipping type to absolute, used the whole validation set during validation, an initial learning rate of 0.0001, batches of ∼8192 tokens, and maximum sentence length of 200 tokens. we have optimized for bleu. custom checkpoint intervals have been used during training, with final systems using between 2 and 11 checkpoints per epoch, consistent within sets of experiments (e.g., vocabulary size sweeping). for finetuning, the checkpoint interval is set to 9, resulting in about 2 checkpoints per epoch for news and 13 for hansard. for finetuning, we used an initial learning rate of 0.00015 (decreasing by a factor of 0.7 if there was no improvement after 8 checkpoints). decoding was done with beam size 5. in the following sections, we describe the experiments that led to our submitted systems. our final systems were trained on a mix of news and hansard data (section 5.1), using joint bpe (section 5.2), bpe-dropout (for en-iu; section 5.3), tagged backtranslation (for iu-en; section 5.4), finetuning (section 5.5), ensembling, and the use of domain-specific models (section 5.6). @xcite when extracting the bpe vocabulary (which we then used consistently for all experiments) and when applying the bpe model, we used a glossary containing the special tokens produced in preprocessing, moses special tokens, and special tags (section 5.4), to ensure they would not be split. ## conclusions and future work we have presented the results of our iu-en and en-iu systems, showing that a combination of bpe-dropout (for en-iu), backtranslation (for iu-en), domain-specific finetuning, ensembling, and hybrid systems produced competitive results. we performed automatic evaluations of the submitted systems in terms of bleu, chrf (popović, 2015), and yisi-1 (lo, 2020). our en-iu system performed best out of the constrained systems in terms of bleu (15.8, +2.5 above the next-best system), chrf (37.9, +1.7 above the next-best), and yisi (82.4, +0.5 above the next-best). our iu-en system performed third-best out of all systems in terms of bleu (28.0, -1.9 below the best system), third-best in terms of chrf (48.9, -2.0 below the best system), and third-best in terms of yisi-1 (92.3, -0.6 behind the best system). @xcite there remains a wide range of future work to be done to improve translation for this language pair. there is still space to improve roman text output in inuktitut, perhaps even as simply as an automatic postediting approach. different subword segmentations (or ones complementary to bpedropout like he et al. ( 2020 )), particularly ones that capture morphological and phonological aspects of inuktitut may also be promising. in terms of adding monolingual data, we expect that improved data selection for backtranslated data (i.e., to increase topic relevance) may be useful, as would additional inuktitut monolingual data. due to time constraints, we were unable to complete bpe-dropout for iu-en systems; we expect this would have resulted in improved performance. domain finetuning remains a challenge given the very small amount of parallel news data available. we did experiment with mixing hansard and news data for finetuning, but, contrary to chu et al. (2017), were unable to outperform news-only systems on news. it may be worth trying approaches designed to prevent catastrophic forgetting in domain adaptation (thompson et al., 2019). the real test, of course, will be human evaluation; are the systems producing output that might be usable, whether for computer aided translation (via postediting or interactive translation prediction) or for use in other downstream applications?