Transformers for Low-Resource Languages: Is Féidir Linn! (2021)

Abstract: The Transformer model is the state-of-the-art in Machine Translation. However and in general and neural translation models often under perform on language pairs with insufficient training data. As a consequence and relatively few experiments have been carried out using this architecture on low-resource language pairs. In this study and hyperparameter optimization of Transformer models in translating the low-resource English-Irish language pair is evaluated. We demonstrate that choosing appropriate parameters leads to considerable performance improvements. Most importantly and the correct choice of subword model is shown to be the biggest driver of translation performance. SentencePiece models using both unigram and BPE approaches were appraised. Variations on model architectures included modifying the number of layers and testing various regularization techniques and evaluating the optimal number of heads for attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus were used for evaluation. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model. Improvements were observed across a range of metrics and including TER and indicating a substantially reduced post editing effort for Transformer optimized models with 16k BPE subword models. Bench-marked against Google Translate and our translation engines demonstrated significant improvements. The question of whether or not Transformers can be used effectively in a low-resource setting of English-Irish translation has been addressed. Is féidir linn - yes we can.

## introduction the advent of neural machine translation (nmt) has heralded an era of high-quality translations. however, these improvements have not been manifested in the translation of all languages. large datasets are a prerequisite for high quality nmt. this works well in the context of well-resourced languages where there is an abundance of data. in the context of low-resource languages which suffer from a sparsity of data, alternative approaches must be adopted. an important part of this research involves developing applications and models to address the challenges of low-resource language technology. such technology incorporates methods to address the data scarcity affecting deep learning for digital engagement of low-resource languages. it has been shown that an out-of-the-box nmt system, trained on english-irish data, achieves a lower translation quality compared with using a tailored smt system @xcite . it is in this context that further research is required in the development of nmt for low-resource languages and the irish language in particular. most research on choosing subword models has focused on high resource languages @xcite . in the context of developing models for english to irish translation, there are no clear recommendations on the choice of subword model types. one of the objectives in this study is to identify which type of subword model performs best in this low resource scenario. ## background native speakers of low-resource languages are often excluded from useful content since, more often than not, online content is not available to them in their language of choice. such a digital divide and the resulting social exclusion experienced by second language speakers, such as refugees living in developed countries, has been well documented in the research literature @xcite . research on machine translation (mt) in low-resource scenarios directly addresses this challenge of exclusion via pivot languages @xcite , and indirectly, via domain adaptation of models @xcite . breakthrough performance improvements in the area of mt have been achieved through research efforts focusing on nmt @xcite . consequently, state-of-the-art (soa) performance has been attained on multiple language pairs @xcite . ## proposed approach hpo of rnn models in low-resource settings has previously demonstrated considerable performance improvements. the extent to which such optimization techniques may be applied to transformer models in similar low-resource scenarios is evaluated as part of this study. evaluations included modifying the number of attention heads, the number of layers and experimenting with regularization techniques such as dropout and label smoothing. most importantly, the choice of subword model type and the vocabulary size are evaluated. in order to test the effectiveness of our approaches, optimization was carried out on two english-irish parallel datasets: a general corpus of 52k lines from the directorate general for translation (dgt) and an in-domain corpus of 88k lines of public administration (pa) data. with dgt, the test set used 1.3k lines and the development set comprised of 2.6k lines. in the case of the pa dataset, there were 1.5k lines of test data and 3k lines of validation. all experiments involved concatenating source and target corpora to create a shared vocabulary and a shared sentencepiece subword model. the impact of using separate source and target subword models was not explored. the approach adopted is illustrated in figure 1 . two baseline architectures, rnn and transformer, are evaluated. on evaluating the hyperparameter choices for transformer models, the values outlined in ## environmental impact motivated by the findings of stochastic parrots @xcite , energy consumption during model development was tracked. prototype model development used colab pro, which as part of google cloud is carbon neutral @xcite . however, longer running transformer experiments were conducted on local servers using 324 gco 2 per kwh foot_1 (seai, 2020). the net result was just under 10 kgco 2 created for a full run of model development. models developed during this study, will be reused for ensemble experiments in future work. ## discussion validation accuracy, and model perplexity, in developing the baseline and optimal models for the dgt corpus are illustrated in figure 4 and figure 5 . rapid convergence was observed while training the baseline model such that little accuracy improvement occurs after 20k steps. including a subword model led to much slower convergence and there were only marginal gains after 60k steps. furthermore, it is observed that training the dgt model, with a 16k bpe submodel, boosted validation accuracy by over 8% compared with its baseline. with regard to the key metric of perplexity, it is shown to rise after training for 15k steps in the baseline models. ppl was observed to rise at later stages, typically after 40k steps in models developed using subword models. perplexity (ppl), shows how many different, equally probable words can be produced during translation. as a metric for translation performance, it is important to keep low scores so the number of alternative translations is reduced. therefore, for future model development it may be worthwhile to set ppl as an early stopping parameter. on examining the ppl graphs of figure 4 and figure 5 , it is clear that a lower global minimum is achieved when the transformer approach is used with a 16k bpe submodel. the ppl global minimum (2.7) is over 50% lower than the corresponding ppl for the transformer base model (5.5). such a finding illustrates that choosing an optimal submodel delivers significant performance gains. translation engine performance was bench-marked against google translate's foot_2 english to irish translation service which is freely available on the internet. four random samples were selected from the english source test file and are presented in the optimal parameters selected in this discovery process are identified in bold in training transformer models with a reduced number of attention heads led to a marginal improvement in translation accuracy with a smaller corpus. our best performing model on a 55k dgt corpus, with 2 heads and a 16k bpe submodel, achieved a bleu score of 60.5 and a ter score of 0.33. by comparison, using 8 heads with the same architecture and dataset yielded 60.3 for the bleu and 0.34 for the ter. in the case of a larger 88k pa corpus, all transformer models using 8 heads performed better than equivalent models using just 2 heads. ## conclusion in our paper, we demonstrated that a random search approach to hyperparameter optimization leads to the development of high-performing translation models. we have shown that choosing subword models, in our low-resource scenarios, is an important driver for the performance of mt engines. moreover, the choice of vocabulary size leads to varying degrees of performance. within the context of low-resource english to irish translation, we achieved optimal performance, on a 55k generic corpus and an 88k in-domain corpus, when a transformer architecture with a 16k bpe submodel was used. the importance of selecting hyperparameters in training low-resource transformer models was also demonstrated. by reducing the number of hidden layer neurons and increasing dropout, our models performed significantly better than baseline models and google translate. performance improvement of our optimized transformer models, with subword segmentation, was observed across all key indicators namely a higher validation accuracy, a ppl achieved at a lower global minimum, a lower post editing effort and a higher translation accuracy.