Machine Translation in the Covid domain: an E nglish- I rish case study for L o R es MT 2021 (2021)

Abstract: Translation models for the specific domain of translating Covid data from English to Irish were developed for the LoResMT 2021 shared task. Domain adaptation techniques, using a Covid-adapted generic 55k corpus from the Directorate General of Translation, were applied. Fine-tuning, mixed fine-tuning and combined dataset approaches were compared with models trained on an extended in-domain dataset. As part of this study, an English-Irish dataset of Covid related data, from the Health and Education domains, was developed. The highestperforming model used a Transformer architecture trained with an extended in-domain Covid dataset. In the context of this study, we have demonstrated that extending an 8k in-domain baseline dataset by just 5k lines improved the BLEU score by 27 points.

## introduction neural machine translation (nmt) has routinely outperformed statistical machine translation (smt) when large parallel datasets are available @xcite . furthermore, transformer based approaches have demonstrated impressive results in moderate low-resource scenarios @xcite . nmt involving transformer model development will improve the performance in specific domains of low-resource languages @xcite . however, the benefits of nmt are less clear when using very low-resource machine translation (mt) on in-domain datasets of less than 10k lines. the irish language is a primary example of a low-resource language that will benefit from such research. this paper reports the results for the mt system developed for the english-irish shared task at loresmt 2021 @xcite . relevant work is presented in the background section followed by an overview of the proposed approach. the empirical findings are outlined in the results section. finally, the key findings are presented and discussed. ## proposed approach figure 1 : proposed approach. optimal hyperparameters are applied to transformer models which are trained using one of several possible approaches. the training dataset composition is determined by the chosen approach. models are subsequently evaluated using a suite of metrics. hyperparameter optimization of recurrent neural network (rnn) models in low-resource settings has previously demonstrated considerable performance improvements @xcite . the extent to which such optimization techniques may be applied to transformer models in similar low-resource scenarios was evaluated in a previous study @xcite . evaluations included modifying the number of attention heads, the number of layers and experimenting with regularization techniques such as dropout and label smoothing. most importantly, the choice of subword model type and the vocabulary size were evaluated. in order to test the effectiveness of our approach, models were trained using three english-irish parallel datasets: a general corpus of 52k lines from the directorate general for translation (dgt) and two in-domain corpora of covid data (8k and 5k lines). all experiments involved concatenating source and target corpora to create a shared vocabulary and a shared sentence-piece @xcite subword model. the impact of using separate source and target subword models was not explored. the approach adopted is illustrated in figure 1 and the datasets used in evaluating this approach are outlined in ## discussion standard transformer parameters identified in a previous study were observed to perform well @xcite . reducing hidden neurons to 256 and increasing regularization dropout to 0.3 improved translation performance and these hyperparameters were chosen when building all transformer models. furthermore a batch size of 2048 and using 6 layers for the encoder / decoder were chosen throughout. the results demonstrate that translation performance for specific domains is driven by the amount of data which is available for that specific domain. it is noteworthy that an in-domain dataset of 13k lines (covid extended), trained for just 35k steps outperformed by 22.1 bleu points the corresponding out-of-domain 52k dataset (dgt) which was trained for 200k steps. ## conclusion and future work in the official evaluation for loresmt 2021, our english-irish system was ranked first according to the bleu, ter and chrf scores. we demonstrate that a high performing in-domain translation model can be built with a dataset of 13k lines. developing a small in-domain dataset, of just 5k lines, improved the bleu score by 27 points when models were trained with the combined covid baseline and custom covid dataset. following on from our previous work, careful selection of transformer hyperparameters, and using a 16k bpe sentencepiece submodel, enabled rapid development of high performing translation models in a low-resource setting. within the context of our research in low-resource english to irish translation, we have shown that augmenting in-domain data, by a small amount, performed better than approaches which incorporate fine-tuning, mixed fine-tuning or the combination of domains. as part of our future work, we plan to develop english-irish mt models trained on a dataset derived from the health domain. domain adaptation, through fine-tuning such models with the covid extended dataset may further improve covid mt performance.