A dam M ickiewicz U niversity’s E nglish- H ausa Submissions to the WMT 2021 News Translation Task (2021)

Abstract: This paper presents the Adam Mickiewicz University’s (AMU) submissions to the WMT 2021 News Translation Task. The submissions focus on the English↔Hausa translation directions, which is a low-resource translation scenario between distant languages. Our approach involves thorough data cleaning, transfer learning using a high-resource language pair, iterative training, and utilization of monolingual data via back-translation. We experiment with NMT and PB-SMT approaches alike, using the base Transformer architecture for all of the NMT models while utilizing PB-SMT systems as comparable baseline solutions.

## introduction we describe the adam mickiewicz university's submissions to the wmt 2021 news translation task. we focused on translation between hausa and english -a low-resource translation scenario between distant languages. our methods combine data cleaning with opusfilter @xcite and fasttext @xcite , transfer learning @xcite , iterative training, and back-translation @xcite . all nmt models were trained with fairseq @xcite , while the first iteration of the back-translation was generated with moses @xcite . the results presented in the paper are based on the first released development set ("dev-1"), which consists of 1000 sentences, the final development set ("dev-full"), which adds additional 1000 sentences to the first development set, and the released test set without additional test suites ("test"). the test set consists of 1000 sentences in @xmath0. the final submissions significantly outperform the vanilla nmt baselines in terms of bleu @xcite metric results, as implemented in sacrebleu @xcite with default settings. all systems were trained in a constrained scenario i.e., using the data provided by the organizers of wmt 2021 only. ## data preparation the quality of the training data has a great impact on the final performance of the nmt models @xcite . the data preparation consisted of data cleaning and filtering performed by using opusfilter @xcite pipelines. we specified separate pipelines for monolingual and parallel data. data cleaning phase consisted of normalizing punctuation, removing non-printable characters, and decoding html entities by using moses @xcite pre-processing scripts. we applied subword segmentation on filtered data by using sentencepiece @xcite tool with byte-pair-encoding (bpe) @xcite algorithm. the corpora we used for model training, along with the number of sentences before filtering, are specified in monolingual data filtering for the monolingual data filtering, we defined an opusfilter pipeline that consists of the following filters: the sentence length filter requires that the sentence contain a minimum of 3 and a maximum data type sentences corpora parallel en-ha 751,560 khamenei, opus, paracrawl monolingual en 41,428,626 news crawl (only 2020) monolingual ha 2,311,959 news crawl, commoncrawl parallel de-en 8,600,361 tilde rapid, commoncrawl, europarl, news commentary, paracrawl parallel data filtering the filters used in the parallel data filtering pipeline are nearly identical to those used in the monolingual data filtering pipeline. filters are applied to both the source and target sentences in this scenario. we also included a length ratio filter with a threshold of 2, indicating that a sentence on the source side can be up to twice as long as a sentence on the target side and vice versa. a similar pipeline was applied to the german-english data that was used for transfer learning. we downsampled 3m sentence pairs from paracrawl due to the imbalance in the german-english data. ## approach our models combine transfer learning from a high-resource language pair (german-english), iterative training, and back-translation. we used fairseq @xcite toolkit in our experiments with nmt models, while we used moses @xcite toolkit for our experiments with pb-smt models. all of our nmt models follow the base transformer architecture @xcite , using relu as the activation function and adam (kingma and ba, 2015) as the optimizer with the following parameters: β 1 = 0.9, β 2 = 0.98, = 1e-8. we set the inverse square root learning rate scheduling with a peak value of 1e-3. we used learning rate warmup stage for 4000 updates with initial learning rate of 1e-7. dropout probability was set to 0.2, while the attention dropout probability was set to 0.1. we also used label smoothing with a value of 0.1. in the case of baseline english-hausa models, the joint vocabulary was based on both english and hausa data. in all cases, the vocabulary size was set to 32,000. the pb-smt models were trained with default settings with moses @xcite toolkit. in addition, we trained a 5-gram operation sequence model @xcite . all language models are 5-gram models and were binarized with kenlm @xcite . the models were trained on tokenized, word-level, lowercased sentences. re-casing was applied to the model outputs. after training the base models, we also applied mert (minimum error rate training) @xcite tuning on the development set. ## final results ## post-submission work due to a lack of computing power and time, our experiments and submissions were based on single model training. after the submission deadline, we retrained the final models three times with different seeds.