Character-level NMT and language similarity (2023)

Abstract: We explore the effectiveness of character-level neural machine translation using Transformer architecture for various levels of language similarity and size of the training dataset. We evaluate the models using automatic MT metrics and show that translation between similar languages benefits from character-level input segmentation, while for less related languages, character-level vanilla Transformer-base often lags behind subword-level segmentation. We confirm previous findings that it is possible to close the gap by finetuning the already trained subword-level models to character-level.

## introduction character-level nmt has been studied for a long time, with mixed results compared to subword segmentation. in the mt practitioner's discourse, it has sometimes been assumed that characterlevel systems are more robust to domain shift and better in the translation of morphologically rich languages. recent studies @xcite show that there are no conclusive proofs for these claims. at the same time, character-level systems have been reliably shown to be robust against source-side noise. in terms of general translation quality, they often either underperform or are on par with their subword-level counterparts @xcite . also, both training and inference speeds are lower and memory requirements are higher due to longer sequence lengths (mostly because of the quadratic complexity of the transformer attention mechanism with respect to the input length @xcite ) unless specialized architectures are used. in this work, we present experiments on a specific use-case of translation of related languages. we train bilingual transformer translation models to translate between czech and croatian, german, hungarian, slovak, or spanish. we vary the training dataset size, vocabulary size and model depth and study the effects. we show that in the baseline configuration with vanilla transformer-base, character-level models outperform subword-level models in terms of automated evaluation scores only in closely related czech-slovak translation pair. finally, we confirm that it is possible to obtain a better quality of the char-level translation for less related languages by first training a subword-level model and in the later stage of the training switching to character-level processing. since they provide a comprehensive overview of the field up to today, we will only very briefly list the most influential works in this section, and refer the reader to the detailed analysis in @xcite . in one of the earliest works, @xcite use rnn with character segmentation on the decoder side. lee et al. (2017) @xcite . @xcite show that problems with slow training and worse final translation quality for character-level nmt models can be largely mitigated by first training with subword segmentation and subsequently finetuning on character-segmented text. however, a problem of lower speed (due to longer sequence length) persists, which can make both the training and inference prohibitively costly and slow, especially for models that make use of a larger context than only one sentence. our work specifically targets character-level translation of closely related languages. in wmt 2019 similar language translation task @xcite , @xcite show that character-level nmt is effective for translation between closely related portuguese and spanish and in multilingual low-resource translation for indo-european languages task at wmt21 @xcite , @xcite successfully apply character-level nmt to translation between catalan and occitan. ## conclusions we trained standard transformer models to translate between languages with different levels of similarity both on subword-segmented and character-segmented data. we also varied the model depth and the training set size. we show that character-level models outperform subwordsegmented models on the most closely related language pair (czech-slovak) as measured by automated mt quality metrics. finetuning models trained with subword-level segmentation to character-level increases the performance in some cases. after finetuning, character-level models surpass the quality of subword-level models also for czech-croatian. other, less similar language pairs reach similar preformances for both subword-and character-level models.