Machine Translation for E nglish– I nuktitut with Segmentation, Data Acquisition and Pre-Training (2020)

Abstract: Translating to and from low-resource polysynthetic languages present numerous challenges for NMT. We present the results of our systems for the English–Inuktitut language pair for the WMT 2020 translation tasks. We investigated the importance of correct morphological segmentation, whether or not adding data from a related language (Greenlandic) helps, and whether using contextual word embeddings improves translation. While each method showed some promise, the results are mixed.

## introduction this paper presents the neural machine translation (nmt) systems submitted by the university of groningen to the wmt 2020 translation task foot_0 between inuktitut and english in both directions (en↔iu), describing both constrained and unconstrained systems where we investigated the following research questions: in section 2 we present the main data and evaluation measures used. in section 3 we present experiments with morphological segmentation methods. section 4 presents the results of our translation systems, and in section 5 we present our conclusions. ## corpora and evaluation the preprocessing followed the procedure of @xcite , carrying out the following steps in order: spelling normalisation and romanisation (only for iu), punctuation normalisation, tokenisation, and truecasing (only for en). parallel data is additionally filtered (ratio 15, minimum and maximum length 1 and 200, respectively). as monolingual data we use the common crawl (cc) corpus for inuktitut, and the 2019 version of newscrawl for english. for cc we also filter out duplicate lines, lines of which more than 10% of the characters are neither alphanumerical nor standard punctuation, and lines that contain more than 200 words. these steps reduce the amount of data considerably, from 164,766 to 28,391 lines. line deduplication is also applied to hansards. 2 since the parallel training data contains only hansards, we used part of the news from the dev set as additional training data by splitting the news part of the dev set: the first 1859 lines are used for training and the last 567 for development. we refer to these subsets as newsdevtrain and newsdevdev, respectively. tables 1 and 2 show the parallel and monolingual datasets, respectively, used for training after preprocessing. during development, we evaluated our systems on the news and hansards portions of the development set, separately. we used two automatic evaluation metrics: bleu @xcite and chrf @xcite . chrf is our primary evaluation metric for @xmath0, due to the fact that this metric has been shown to correlate better than bleu with human evaluation when the target language is agglutinative @xcite . bleu is our primary evaluation metric for @xmath1, as the correlations with human evaluation of bleu and chrf are roughly on par for en as the target language. prior to evaluation the mt output is detruecased (only en) and detokenized with moses' scripts. ## segmentation with intrinsic evaluation like many polysynthetic languages, inuktitut has a high degree of inflection and agglutination, leading to very long words with a very high morpheme-toword ratio @xcite . by our estimation, inuktitut has an average of around 4.39 morphemes per word. this means on average there are more potential boundaries, as well as more actual segmentation boundaries to locate per word, making segmentation particularly challenging. inconsistent segmentation harms an nmt model's ability to extract knowledge, because it reduces the frequency and activation of all vocabulary items during training, such that for each individual element in the vocabulary is found in fewer contexts. at inference, inconsistent segmentation can result in morphs that are out-of-vocabulary, resulting in information loss. we hypothesize that linguistically correct segmentation may be particularly beneficial for translation with polysynthetic languages because it could provide more consistent isolation of concepts into subwords. we evaluated a broad pool of segmenters to determine how close various methods can achieve linguistically correct segmentation, comparing results to reference segmentations obtained from the inuktitut computing github repository foot_2 . this repository contains 1096 inuktitut words, manually segmented at the national research council of canada (nrc). our experiments include: rule-based with uqailaut foot_3 ; morfessor baseline (semi-supervised) @xcite ; morfessor flatcat (semisupervised) @xcite ; lmvr (unsupervised) @xcite ; and neural transformer segmentation (supervised). we used uqailaut's rule-based segmenter to create additional annotated segmentations used to train the supervised and semi-supervised systems. in total 600,000 segmentations of unique words from the hansard training dataset were created. all semisupervised and unsupervised systems were trained with the hansard training corpus. for training semisupervised methods, we use 60,000 of the collected segmentations with uqailaut as annotated training data, and another 3,000 as validation data. for lmvr we set the maximum lexicon size to 20,000. related to our work, a previous study @xcite compared segmentation methods based on their ability to generate linguistically correct segmentations for several low-resource mexican polysynthetic languages. their proposed rnn-based neural approach outperformed baselines of other common approaches, so we also tested a neural segmentation method, but instead of an rnn we use a transformer architecture. we implement this neural segmenter using marian foot_4 . on the source side, the unsegmented words are used as input data. the corresponding segmented words are used as target data. on the target side we denote the segmentation boundary by adding a boundary token (@), like in the following example: source: a k i r a q t u q t u t target: a k i r a q @ t u q @ t u t we trained three neural segmentation models: one on all 600,000 annotated segmentations, plus two with 45,000 annotated segmentations, one with only unambiguous annotations foot_5 and one with a random selection from the pool of 600,000. unfortunately, both the neural and rule-based models sometimes fail to segment the input word. this makes them unfit to use in a translation system; since some words are left unsegmented, and this leads to a very large vocabulary size which hurts the translation performance. @xcite previously explored improving the coverage of the uqailaut morphological analyser with the use of an rnn based approach. in @xcite , an srnn extension to the uqailaut morphological analyzer is used in an smt system, and yields a statistically significant improvement for @xmath2. similar to their approach, we combined the best performing models of the intrinsic evaluation, to construct a custom 3-step segmenter to improve the coverage. this method initially applies the rule-based segmenter. preliminary experiments with this approach still resulted in a very large vocabulary size. to reduce the vocabulary size further and combine all steps into a single model, afterwards we perform vocabulary reduction using lmvr. we specify a lexicon size of 20,000, which results in an actual vocabulary size of 41,024. the vocabulary reduction applied to the 3-step model leads to a drop in f1 and accuracy. this could be either because the vocabulary reduction leads to fewer segmentation boundaries per word, or because lmvr changes the model too much. ## translation experiments unless mentioned otherwise, the translation models are trained using marian (junczys-dowmunt et al., 2018) v1.9.0 on an nvidia v100. the translation models use the transformer model type with default settings. we use the ce-mean-words cost function. we perform a validation run every 5,000 update steps and apply early stopping after the validation cost stalls 5 times in a row. the model with the best translation score on the validation set (section 2) is stored for each experiment. ## conclusions this paper has reported on the systems submitted by the university of groningen to the english↔inuktitut translation directions of the news shared task at wmt 2020. 15 our best results were obtained using well-established techniques, including oversampling domain-specific training data, backtranslation, tags, fine-tuning and ensembling. the use of morphological segmentation (rq1) led to results that were on par with those obtained by bpe in terms of automatic evaluation metrics. one problem is that existing morphological segmenters for low-resourced languages like inuktitut suffer from poor coverage, which impedes making a complete comparison with more automatic methods. the extrinsic comparisons between segmenters showed that a more accurate morphological segmentation does not lead to improved translation performance. we further found that existing language agnostic segmenters struggle to produce correct segmentations on inuktitut, and that neural methods appear to be more suitable for polysynthetic languages (cf. @xcite ) . note also the importance of limiting the vocabulary size of morphological segmentation for mt, which could be explored further. the use of additional data from inuktitut did improve the results slightly, but not the addition of data from a related language, greenlandic (rq2). the fact that its usefulness was limited could be due to the fact that half of the test set was from a specific domain for which considerable amounts of data were already available to train (hansards). finally, the use of contextual embeddings (rq3), led to mixed results since it resulted in an improvement for one direction but a degradation for the other.