Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for M altese (2024)

Abstract: In Machine Translation, various tokenisers are used to segment inputs before training a model. Despite tokenisation being mostly considered a solved problem for languages such as English, it is still unclear as to how effective different tokenisers are for morphologically rich languages. This study aims to explore how different approaches to tokenising Maltese impact machine translation results on the English-Maltese language pair.We observed that the OPUS-100 dataset has tokenisation inconsistencies in Maltese. We empirically found that training models on the original OPUS-100 dataset led to the generation of sentences with these issues.We therefore release an updated version of the OPUS-100 parallel English-Maltese dataset, referred to as OPUS-100-Fix, fixing these inconsistencies in Maltese by using the MLRS tokeniser. We show that after fixing the inconsistencies in the dataset, results on the fixed test set increase by 2.49 BLEU points over models trained on the original OPUS-100. We also experiment with different tokenisers, including BPE and SentencePiece to find the ideal tokeniser and vocabulary size for our setup, which was shown to be BPE with a vocabulary size of 8,000. Finally, we train different models in both directions for the ENG-MLT language pair using OPUS-100-Fix by training models from scratch as well as fine-tuning other pre-trained models, namely mBART-50 and NLLB, where a finetuned NLLB model performed the best.

## introduction tokenisation tends not to be given much attention in machine translation (mt), particularly since it is thought of as a solved problem for languages such as english @xcite . however, it can become an issue when generic tokenisers are applied to languages that have particular syntactic rules that are different from those commonly found in english. this is reflected by @xcite who demonstrate that different tokenisers affect language pairs very differently. moreover, several modern neural machine translation approaches also employ subword tokenisation in conjuction with word tokenisers as part of the preprocessing step @xcite @xcite . not much thought tends to be given to the impact that tokenisation could have on the results of the machine translation. it is well known however, that tokenisation affects bleu results @xcite . in our research, we investigate the impact that different tokenisation approaches can have on the mt evaluation of maltese, a morphologically rich language with semitic roots. we experiment with different tokenisers, namely bpe @xcite , sentencepiece @xcite , moses tokeniser @xcite , opennmt tokeniser @xcite , and a regex tokeniser specifically built for maltese @xcite . we refer to this as the mlrs tokeniser. it was seen that most regular word-level english tokenisers do not tokenise/detokenise everything correctly. for example, it was seen that if the mt output has characters such as "-", an english detokeniser usually splits this character from other words by adding a space. this is the correct approach for english, but in maltese most articles contain "-" and should remain joined as one word in the output (such as il-kelb, (the dog) should be tokenised as il-kelb rather than il -kelb. in order to carry out an empirical evaluation of different tokenisation approaches, we also consider different machine translation (mt) approaches, including a baseline system trained from scratch using the base transformer architecture @xcite , a model based on the large transformer architecture, a fine-tuned mbart-50 model @xcite , nllb (outof-the-box) @xcite and a finetuned version of nllb, referred to as nllb-ft. notably, the pre-trained model mbart-50 does not contain previous knowledge of maltese whereas nllb has encountered maltese during its training. another challenge that we face when training nmt systems for low-resource languages is the lack of high-quality publicly available data. for our experiments, we utilize the opus-100 dataset @xcite . however, upon further analysis, we noticed that the maltese documents contained several tokenisation errors that added spacing where there should be none. these errors do not occur in a systematic way, which motivated us to look into the impact that such inconsistencies would have on the results. evaluation metrics such as bleu, tend to favour a larger number of n-grams. the type of errors present in opus-100 were producing more tokens and this could result in an inflated bleu score. to this end, we investigate the impact of incorrect tokenisation in opus-100 when maltese is the target language. our translation efforts focus on the english -maltese language pair since english is normally used as the source language whenever the target is maltese, both in a research setting but also in an application scenario. our contributions are the following: ## conclusion this paper presents an updated version of opus-100, opus-100-fix, which fixes numerous inconsistencies in the maltese data. it is seen that by fixing these inconsistencies, the results improve. apart from that, we also experiment with numerous tokenisers where we observed that using bpe alone, with a vocabulary size of 8,000, achieves the best results on our data. we finally experiment with different models, both fine-tuned (including nllb and mbart) and those trained from scratch, and it can be seen that fine-tuning nllb yields the best performance.