Surprise Language Challenge: Developing a Neural Machine Translation System between P ashto and E nglish in Two Months (2021)

Abstract: In the media industry and the focus of global reporting can shift overnight. There is a compelling need to be able to develop new machine translation systems in a short period of time and in order to more efficiently cover quickly developing stories. As part of the EU project GoURMET and which focusses on low-resource machine translation and our media partners selected a surprise language for which a machine translation system had to be built and evaluated in two months(February and March 2021). The language selected was Pashto and an Indo-Iranian language spoken in Afghanistan and Pakistan and India. In this period we completed the full pipeline of development of a neural machine translation system: data crawling and cleaning and aligning and creating test sets and developing and testing models and and delivering them to the user partners. In this paperwe describe rapid data creation and experiments with transfer learning and pretraining for this low-resource language pair. We find that starting from an existing large model pre-trained on 50languages leads to far better BLEU scores than pretraining on one high-resource language pair with a smaller model. We also present human evaluation of our systems and which indicates that the resulting systems perform better than a freely available commercial system when translating from English into Pashto direction and and similarly when translating from Pashto into English.

## introduction the horizon 2020 european-union-funded project gourmet foot_0 (global under-resourced media translation) aims to improve neural machine translation for under-resourced language pairs with a special emphasis on the news domain. the two media partners in the gourmet project, the bbc in the uk and deutsche welle (dw) in germany, publish news content in 40 and 30 different languages, respectively, and gather news in over 100 languages. in such a global information scenario, machine translation technologies become an important element in the everyday workflow of these media organisations. surprise language exercises @xcite started in march 2003, when the us defense advanced research projects agency (darpa) designated cebuano, the second most widely spoken indigenous language in the philippines, as the focus of an exercise. teams were given only ten days to assemble language resources and to create whatever human language technology they could in that time. these events have been running annually ever since. the gourmet project undertook its surprise language evaluation as an exercise to bring together the whole consortium to focus on a language pair of particular interest to the bbc and dw for a short period of time. given the impact of the covid-19 pandemic, a twomonth period was considered realistic. on 1 february 2021, bbc and dw revealed the chosen language to be pashto. by completing and documenting how this challenge was addressed, we prove we are able to bootstrap a new high quality neural machine translation task within a very limited window of time. there has also been a considerable amount of recent interest in using pretrained language models for improving performance on downstream natural language processing tasks, especially in a low resource setting @xcite @xcite , but how best to do this is still an open question. a key question in this work is how best to use training data which is not english (en) to pashto (ps) translations. we experimented, on the one hand, with pretraining models on a high-resource language pair (german-english, one of the most studied high-resource language pairs) and, on the other hand, with fine-tuning an existing large pretrained translation model (mbart50) trained on parallel data involving english and 49 languages including pashto @xcite . we show that both approaches perform comparably or better than commercial machine translation systems especially when pashto is the output language, with the large multilingual model achieving the highest translation quality between our two approaches. the paper is organised as follows. section 2 motivates the choice of pashto and presents a brief analysis of the social and technical context of the language. section 3 describes the efforts behind the crawling of additional monolingual and parallel data in addition to the linguistic resources already available for english-pashto. section 4 introduces the twofold approach we followed in order to build our neural machine translation systems: on the one hand, we developed a system from scratch by combining mbart-like pretraining, german-english translation pretraining and fine-tuning; on the other hand, we also explored fine-tuning on the existing pretrained multilingual model mbart50. we present automatic results and preliminary human evaluation of the systems in section 5. ## the case for pashto the primary goal, when selecting which low-resource language pair to work on, was to provide a tool that would be useful to both the bbc and deutsche welle. it had to be an under-resourced language with high news value and audience growth potential, and one that could pose a satisfactory research challenge to complement the wider goals of the project. pashto ticked all of these boxes. pashto is one of the two official languages of afghanistan along with dari. almost half of the country's 37.5 million people, up to 10 percent of the population in neighbouring pakistan, and smaller communities in india and tajikistan speak pashto, bringing estimates of pashto speakers worldwide around 45-50 million @xcite . europe hosts a growing number of pashto speakers, too. as of the end of 2020, there were over 270,000 afghans living in ger-many foot_1 and 79,000 in the uk foot_2 . projecting from afghanistan's national linguistic breakdown, foot_3 up to half of these could be pashto speakers. pashto (also spelled pukhto and pakhto is an iranian language of the indo-european family and is grouped with other iranian languages such as persian, dari, tajiki, in spite of major linguistic diferences among them. pashto is written with a unique enriched perso-arabic script with 45 letters and four diacritics. translating between english and pashto poses interesting challenges. pashto has a richer morphology than that of english; the induced data sparseness may partly be remedied with segmentation in subword units tokenization models such as sentencepiece @xcite , as used in mbart50. there are pashto categories in pashto that do not overtly exist in english (such as verb aspect or the oblique case in general nouns) and categories in english that do not overtly exist in pashto (such as definite and indefinite articles), which may pose a certain challenge when having to generate correct text in machine translation output. due to the chronic political and social instability and conflict that afghanistan has experienced in its recent history, the country features prominently in global news coverage. closely following the developments there remains a key priority for international policy makers, multilateral institutions, observers, researchers and the media, alongside the wider array of individual news consumers. pashto features in bbc monitoring's language portfolio. enhancing the means to better follow and understand pashto resources first hand through machine translation offers a valuable contribution. the interest of commercial providers of machine translation solutions in pashto is recent and there is room for improvement for existing solutions. google translate integrated pashto in 2016, ten years after its launch. 5 amazon followed suit in november 2019 and microsoft translator added pashto into its portfolio in august 2020. 6 nevertheless, pashto has been of interest to the gourmet media partners long before ## data creation the process of data collection and curation is divided into two clearly different processes to obtain: (a) training data, and (b) development and test data. this section describes these two proceses. note that our neural systems were trained with additional training data which will be described in section 4. ## training of neural machine translation systems we developed two different neural models: a from-scratch system, and a larger and slower system based on an existing pretrained model. the development of the former starts with a mediun-size randomly-initialized transformer @xcite , whereas the latter is obtained by fine-tuning the larger downloadable mbart50 pretrained system @xcite . remarkably, mbart50 has been pretrained with some pashto (and english) data which makes it a very convenient model to explore. the size of pretrained models make them poor candidates for production environments, especially where they are required to run on cpu-only servers as it is the case in the gourmet project, yet translations have to be available at a fast rate. in those scenarios, the from-scratch system may be considered a more efficient alternative. our mbart50 systems can still be useful in those scenarios to generate synthetic data with which to train smaller models. ## results and discussion tables 2 and 3 show bleu and chrf2 scores, respectively, for the english to pashto systems with different test sets. the evaluation metrics for the google mt system are also included for reference purposes. similarly, tables 4 and 5 show bleu and chrf2 scores, respectively, for the pashto to english systems. all the scores were computed with sacrebleu @xcite . the test sets considered are the two in-house parallel sets created by bbc and dw (see section 3) and the devtest set provided in the flores foot_11 benchmark (2,698 sentences). as can be seen, the from-scratch system provides worse results than the mbart50-based model obtained after the three-step fine-tuning procedure, which may be easily explained by the smaller number of parameters and the lack of initial knowledge. regarding the mbart50-based models, for the @xmath1, the scores obtained with the non-fine-tuned models for the flores test set are considerably higher than those corresponding to the bbc and dw test sets, which suggests that either they belong to different domains, or they contain very different grammatical or lexical structures, or the flores corpus was used to pretrain mbart50. this indicates that fine-tuning could provide a twofold benefit: on the one hand, it may allow the model to focus on our two languages of interest, partially forgetting what it learned for other languages; on the other hand, it may allow the model to perform domain adaptation. in the @xmath2, except when the last model is evaluated against the flores devtest set, which makes sense as the development set belongs to the domain of the bbc and dw test sets. notably, the system resulting from the three-step fine-tuning process improves google's scores as of april 2021. in the @xmath3, the same trend can be observed, although in this case the best mbart50-based system is noticeably behind the scores of google's system, yet it still provides scores higher than those for the other translation direction. ## conclusion we present a description of our rapid pashto↔english machine translation system building exercise. we performed extensive crawling and data cleaning and alignment, combined with pretraining experiments to deliver a strong translation system for a low-resource language. we test different transfer learning approaches and show that large, multilingual models perform better than smaller models from a high-resource language pair. the data 21 , models 22 and tools 23 are shared publically.