Language Adapters for Large-Scale MT : The GMU System for the WMT 2022 Large-Scale Machine Translation Evaluation for A frican Languages Shared Task (2022)

Abstract: This report describes GMU’s machine translation systems for the WMT22 shared task on large-scale machine translation evaluation for African languages. We participated in the constrained translation track where only the data listed on the shared task page were allowed, including submissions accepted to the Data track. Our approach uses models initialized with DeltaLM, a generic pre-trained multilingual encoder-decoder model, and fine-tuned correspondingly with the allowed data sources. Our best submission incorporates language family and language-specific adapter units; ranking ranked second under the constrained setting.

## introduction there has traditionally been a significant concentration of machine translation research on a few languages -usually indo-european @xcite . data scarcity has hindered the progress of many languages, many with millions of speakers @xcite . the shared task and our submission aim to reverse the trend by focusing on low-resource african languages that have been traditionally ignored by mainstream research. our submission leverages different approaches to produce a multilingual mt system that can handle all 26 languages covered by the shared task: we expand on each of these components in our system description and the related work section. our deltalm model was fine-tuned in the first step using parallel data collected from all 26 languages. after fine-tuning the previous model, we adapter-tune the language-specific adapters. our third step is to adapter-tune the family-specific and sub-family-specific adapters based on the previous adapter-tune model. we submit the second and third models as our submissions to the shared task. ## evaluation results we use the dev and the hidden test set of the flo-res200 @xcite @xcite benchmark as our validation set and test set respectively. a beam search strategy with a beam size of 5 is used during inference in order to generate target sentences. based on the loss on the validation set, we select the best checkpoint for evaluation. we report bleu, chrf++, and sentencepiece-based bleu using spbleu scores. our model using language-specific adapters significantly outperforms the fine-tuning model. our model with family-specific adapters does not seem to outperform the language-specific adapters on average. but we do obtain some gains for avg @xmath0. going deeper to the results, we do find significant gains for some individual language pairs: for instance, for tswana-english (tsn-eng) we obtain a 1.0 bleu point gain, and for english-hausa (eng-hau) this model is better by 1.2 bleu points. in addition, our phylogeny-inspired adaptors follow a pre-defined path along the trees. this is perhaps too rigid, especially for communities that use a lot of code-switching, or for creole languages and pidgins that are the result of language contact. in future work, we will explore ways to learn the path through the tree, or allow for soft sharing of parameters through attention or mixture of experts units. ## conclusion this paper describes gmu's submission to the large-scale machine translation for african languages of the wmt22 shared task. here we explore if pre-trained models can be useful even for languages on which they have not been pre-trained. our multilingual adapter-tuning translation model, built on deltalm, achieves substantial improvements over simply fine-tuning deltalm. we further try to enhance the model performance with adapter-tuning using phylogeny information. as a result, our submitted systems rank third on the data-constrained track. avg npl denotes the average score of language directions where no language was present in the pre-training of deltalm. avg pl denotes the average score of language directions where at least one language was present in the pre-training of deltalm. avg npl-source denotes the average score of language directions where the source language was not present in the pre-training of deltalm. avg pl-source denotes the average score of language directions where the source language was present in the pre-training of deltalm. avg npl-target denotes the average score of language directions where the target language was not present in the pre-training of deltalm. avg pl-target denotes the average score of language directions where the target language was present in the pre-training of deltalm.