Are Character-level Translations Worth the Wait? Comparing B y T 5 and m T 5 for Machine Translation (2024)

Abstract: Pretrained character-level and byte-level language models have been shown to be competitive with popular subword models across a range of Natural Language Processing tasks. However, there has been little research on their effectiveness for neural machine translation (NMT), particularly within the popular pretrain-then-finetune paradigm. This work performs an extensive comparison across multiple languages and experimental conditions of character- and subword-level pretrained models (ByT5 and mT5, respectively) on NMT. We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. In our analysis, we show how character models’ gains in translation quality are reflected in better translations of orthographically similar words and rare words. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5, suggesting an ability to modulate word-level and character-level information during generation. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.

## introduction character-level and byte-level models foot_0 have been a source of interest in natural language processing (nlp) for many years, with the promise of tokenization-free systems able to process and generate text on a finer granularity. however, these systems have failed to become the dominant paradigm over subword-level models, despite comparable performances. this is likely because of the additional time and compute resources required, due to the longer input sequences used in character-based approaches. there are cases where character models have been shown to outperform subword models. however, these instances may be seen as niche (e.g., tasks that specifically require character information) or unrealistic (e.g., using data corrupted with synthetic noise @xcite ). additionally, previous studies presented only limited evaluations of these systems for popular tasks where character-level information could lead to major performance benefits. in this work, we conduct a comprehensive comparison of character and subword pretrained models on machine translation (mt). despite the popularity of nmt, pretrained character models have not yet been thoroughly assessed for this task, with recent research on character models for mt focusing on models trained from scratch @xcite . we posit that these approaches can only reliably assess translation performance for high-resource languages, where the beneficial effects of multilingual pretraining were found to be less impactful @xcite . character models can leverage more finegrained information, which could be helpful in many challenging nmt settings, such as lowresource translation. to validate this, we fine-tune the character model byt5 @xcite and its subword counterpart mt5 @xcite for translation, for a variety of languages and settings. among our findings, those standing out are: figure 1: byt5's overall source vs. target contribution for the @xmath0: ''i study international relations.'' $: end-of-sentence token. peaks in source importance at the beginning of each word suggest word-level modeling. our findings support the idea that in several realistic circumstances, and particularly for lowresource scenarios, character models are superior in terms of translation quality over their widely used subword counterparts. in our analysis, we further show how these gains in quality connect to specific word properties, such as orthographic similarity and frequency. ## related work character-level models have long been of interest for use in machine translation, dating back to when statistical models were the dominant paradigm @xcite . at that time, character models were already competitive with word-level models, especially when training data was limited @xmath1. @xcite also showed that character models were particularly adept at translating closely related languages such as hindi and urdu. we note that, at the time, subword tokenizers such as bpe @xcite were not yet commonly used. neural approaches to character-level mt were first based on rnns @xcite , with extensive work being done to compare character models to subword models (now equipped with bpe). for nmt, it was shown that character models using an rnn architecture (with and without a cnn for processing characters) perform equally to or better than subword models @xcite @xcite . their better translation performance could be attributed to their ability to handle complex morphology, rare or unseen words, and noisy input @xcite @xcite @xcite . however the inefficiency of character models was already apparent, as they were slower than subword models by a considerable margin @xcite . work had also been done in comparing byte and character level models, with little differences found, however these were mostly experimenting with latin-scripted languages @xcite . more recent work has looked at transformers on the character and byte-level. xue et al. (2022) created byt5 and compared it to its subword counterpart, mt5 @xcite , which are also the models we focus on in this work. their comparisons were however focused on either multilingual classification tasks or english-based generative tasks, but no multilingual generative tasks or machine translation. previous work analyzed character-level transformers trained from scratch for nmt, not using the t5-based models. @xcite looked at ''vanilla'' character models (those without any compression of the sequence length prior to the computation in the transformer), as well as the methods of @xcite , tay et al. (2022), and @xcite for compressing sequence length. they conclude that these character-level models do not provide any benefits over subword models while being less efficient. there are two factors to note with these conclusions. firstly, their experiments show similar performance, but they only experiment on high-resourced languages. given prior work in rnns mentioned above, this does not appear to be the application that would benefit most from character-level models. secondly, they introduce a two-step decoder to achieve character-level decoding from subwordlevel hidden states, however as @xcite points out, this decoder does not scale well to higher-resourced scenarios. this two-step decoder adds an additional layer of complexity to evaluating such models, as an ablation of the model's granularity and the model's decoding process would be necessary to fully understand the performance of each model. added to the fact that there are no pretrained models using this sequence length compression that are comparable to mt5 and byt5 in terms of data used for pretraining or model scale, we do not experiment with the models which compress the sequence length in our work. in the context of low-resource mt, @xcite showed that character-level models can outperform subword models on the lowresource pair xhosa-zulu. @xcite showed that character-level models create higher-quality translations in synthetic low-resource settings of @xmath2, finnish} with a corpus size of 50k parallel sentences. carrión-ponz and casacuberta (2022) showed that quasi-character models (subword models with a small vocabulary of size 350) produce higher-quality translations than subwords with a more standard vocabulary size of 32 thousand when data is limited for a number of european languages, finding consistent improvements across several domains. there are two major caveats with this previous work that should be considered, however. first, previous work using character models for mt focused on training models from scratch, as this is a long-standing practice in the mt field. however, this practice could be especially harmful for low-resource languages, where the paradigm of fine-tuning a pretrained, multilingual model was shown to be effective @xcite . the second caveat is that previous evaluations of cross-lingual transfer were also limited by a relatively small model size (<70m parameters). in contrast, this work evaluates models up to 1.2b parameters. with an order of magnitude more parameters, we investigate the presence of emergent properties of larger character and subword-based nmt models, following the evidence from other generative tasks. within the realm of the extensive previous work done on character models, this work serves to give an updated overview on the performance of character versus subword models for nmt. to our knowledge, we provide the first of such overviews using multilingual, transformer-based models of up to 1.2b parameters. we fine-tune a total of 162 models (varying languages, amount of training data, model size, and model type), and test on 200 languages to thoroughly compare the performance of character and subword models. we also perform an attribution analysis to gain a deeper understanding of the differences between the two granularities of character-level and subword-level, which to our knowledge, has not yet been researched. ## method as our experiments are aimed to provide a fair comparison of character and subword pretrained models for translation, we first justify our model choice, followed by our training scheme, and lastly our choice of metric for evaluation. ## direct translation results our direct translation setup evaluates models on the same language pairs for which they are finetuned. since character models generally outperform subword models, especially when training data is scarce (as noted in section 2), we vary the amount of fine-tuning data, confirming these findings for {german, russian}↔english translation. varying the amount of fine-tuning data reveals the largest difference in the translation quality of character and subword models. figure 2 shows that byt5 outperforms mt5 in all resource levels according to chrf++, and is competitive with mt5 according to comet. when resources are limited, the quality gap between byt5 and mt5 also tends to increase. we see that model size also plays a role, with the large model having the largest quality gap of up to 10 chrf++ points when only 400 sentences are available. this goes against our assumption that, given the differences in architecture being largest between the small models, we would see the largest difference in performance from them (see section 3.1). ## attribution analysis in translation studies, words are widely recog nized as the minimal unit of translation @xcite . it is therefore natural to assume that a source-to-target word or subword mapping might be easier to create than a character-level one, except perhaps for closely related languages where many words translate into orthographically similar cognate words. therefore, it seems that (sub)word models are naturally more suited towards machine translation, but our results from the previous sections appear to contradict that. this naturally leads us to investigate whether and how character models can learn to incorporate word-level information to improve their translation capabilities. a common way to quantify the influence of inputs in driving model predictions is by means of feature attribution methods @xcite , which have previously been applied in the mt domain to highlight word alignments, coreference resolution capabilities, and model training dynamics @xcite @xcite . for our analysis, we study source versus target character contributions in byt5 models using gradients @xcite , which were recently shown to be more faithful than other advanced approaches for language tasks @xcite . one can take gradients with respect to the probability of a predicted token and propagate them back to the model's inputs in order to quantify the importance of input tokens to the resulting prediction. our analysis is inspired by @xcite , where a similar analysis was conducted on subword mt models to highlight a ''language modeling regime'' where high importance is given to the target prefix while disregarding source tokens. in our study, we compute gradients for each input token (i.e., a character) with respect to the next token's prediction probability at every generation step using the inseq library @xcite . for every generated token, gradient vectors are aggregated at the character level using the l2 norm of the gradient vector to gauge the influence of the source and target context on the model's predictions. 8 we verify our hypothesis that character-level nmt models might implicitly operate at a word level by comparing source-side and target-side contributions across different model sizes and language pairs. in this context, a character-bycharacter translation would be characterized by relatively uniform source attributions across all generated characters, while a word-by-word translation would imply a higher source contribution of the characters marking the beginning of a new word, followed by a shift of importance towards the target prefix indicating that completing the word requires limited access to source-side information. figure 1 provides an example of source and target contributions, showing a marked increase in source importance at the beginning of each word. how is word-level information used in character models? figure 7 (left) confirms the decline in source importance throughout generation observed by voita et al. (2021a) for several byt5 models. in the left plot, we compute the average source importance with respect to the character's position in the sentence. while progressing through the generation, we observe that the model relies less on the source side and more on the target side for its next-byte predictions. this aligns with voita et al.'s (2021a) findings for subwordlevel mt models, with the intuitive explanation that a longer generated context can provide more cues for the model to produce a coherent output without the need to attend to source-side tokens. in the right plot, instead, we compute the relative average source importance for byte positions within words. since bytes occurring later in a word also occur later in a sentence, we would expect source importance to drop later in a word as a consequence of the trend seen in the left plot. to disentangle the effect of the in-word byte position from the sentence-level one, we normalize the source importance scores by their average for their corresponding position in the sentence (i.e., we divide every in-word score by the respective in-sentence average from the left plot). as a result, we would expect stable relative source importance close to 100% across all in-word byte positions if the position of characters inside the word did not affect source importance. instead, we observe a rapid decline of source importance within the word across all languages, implying an increase in target-side importance for non-wordinitial generated characters. upon manual inspection of the importance patterns, we note that the target-side importance is focused mainly on the previous characters belonging to the same word, suggesting the usage of word-level information to facilitate translation. interestingly, for @xmath4. the observed trend reflects the utf-8 cyrillic encoding using 2 bytes per character, foot_6 with peaks at first-byte boundaries pointing to the model's ability to discriminate characters and bytes in languages using multi-byte characters. we also note that relative source attributions seem to converge to 100% around the third character for latin script languages and around the sixth in-word character for russian. these results indicate that similar importance patterns might emerge across different languages in character models despite separate fine-tuning procedures. how does resourcedness affect word-level modeling? in figure 8 , we examine how the figure 8 : left: source contribution across word's characters for a byt5-large fine-tuned on @xmath5.4k to 250k). right: source contributions for same-language (@xmath6. model. ## effect of word similarity and frequency character models have previously been shown to have a more robust vocabulary and are thus able to form useful biases for translating orthographically similar words, as well as rare words @xcite . we revisit these findings to see if they hold with larger, multilingually pretrained transformers. can character models exploit word similarity? we proceed to investigate whether character models can operate on character level when desirable. to this end, we focus our analysis on orthographically similar words (osws), starting from the assumption that a character model can easily exploit their similarity by learning to copy matching substrings. we start by assessing whether character models show improved performances for osw translation. we use awesome @xcite to align source, reference, and hypothesis sentences and calculate word-level translation accuracy. our definition of osws is based on the inverse normalized-levenshtein distance of the source and reference words, varying the threshold from 0% to 100%, for example, german gesundheit and dutch gezondheid have a similarity of 70% since they differ in three letters. figure 9 shows the accuracy difference for the large byt5 and mt5 models trained for @xmath7. we observe that, as words become more similar, the accuracy delta also increases in favor of byt5, especially when less fine-tuning data is used. these results indicate that character models can learn osw translations more rapidly and effectively than their subword counterparts. we proceed by examining the source contributions of osws and non-osws using byt5, and how those change according to the number of examples used for fine-tuning. we restrict our analysis @xmath8, @xmath9. we consider only the source importance over characters belonging to the same word, in order to investigate the presence of copying patterns for osws. as shown in figure 10 , the importance of source text grows with the amount of fine-tuning data when translating osws in the @xmath10, suggesting an improved ability of the model in copying information from the source. conversely, we observe a declining trend for the zero-shot @xmath11, converging to the source importance of limburgish, which was not present in the model's pretraining data. this could indicate that the dutch knowledge acquired during pretraining is progressively lost when fine-tuning on increasingly more german data, supporting the results of figure 3 . for non-osws, we find a relative source importance of 94% ± 4% across all three directions and all amounts of training examples, indicating that the observed copying behavior is restricted to highly similar words only. to further confirm this copying behavior is indeed distinct between a subword and character model, we create a synthetic control test set by replacing all proper noun pairs (word pairs tagged as proper nouns in both source and reference target sentences foot_7 ) with random strings of latin alphabet characters matching the original words' length. this is done for our @xmath12, modifying the @xmath13. we observe that for the subword-level mt5 model, the copying mechanism is learned progressively during fine-tuning. meanwhile, the byt5 model instead achieves superior copying performances with very little supervision, achieving 91% accuracy with only 400 training examples. we also observe that mt5's performance on chrf++ improves with increasing examples, while our control set only targets proper nouns, this copying behavior may be far more pervasive, as it could be present in any loan words, regardless of part-of-speech. the large difference in scores, particularly on the models fine-tuned on only 400 examples, seems to show a major difference in the default behavior of the two models. can character models translate rare words more accurately? prior work has shown character models achieving better translation accuracy of rare words @xcite , so we expect the same to be true for large, pretrained, transformers as well. we adopt the same word-level translation accuracy method as earlier in the section, but instead binning words based on their frequency in the english fine-tuning data. we evaluate word-level accuracy of byt5 versus mt5-large when fine-tuned on 10k @xmath14. we additionally distinguish between accuracy using the original source language and accuracy using zero-shot languages. as shown in figure 11 , byt5 has a higher per-word translation accuracy across all frequencies for the zero-shot languages, and all but the most frequent words for german. generally, as the frequency of the word increases, model disparity decreases. this is expected based on previous work @xcite . the disparity is higher for zero-shot languages overall, which indicates a higher level of robustness to changes in vocabulary from byt5, a phenomenon also previously observed @xcite . ## efficiency although we have shown that the translation quality of character models is competitive or better than subword models, another important aspect is the efficiency of character models. while the lack of efficiency of character-level transformer models is well-documented @xcite @xcite , we have yet to see their efficiency on larger (300m+ parameters) transformers for nmt, and so we provide our findings for completeness. we report model training and inference times for 1 nvidia v100 32gb gpu in the slower speed comes largely from the increase in sequence length. while we tried to balance the size of the batches such that each model sees the same amount of text per batch, achieving this required the character models to accumulate gradients for 4 times as many iterations as the subword models. thus, if training or inference speed is a concern, subword models are likely the superior choice, particularly for high-resource languages. in the low-resource setting, there is a significant trade-off between accuracy and speed when choosing between character and subword models. it should be noted that there are several alternatives to vanilla character or byte-level models which offer a speedup, typically by compressing the sequence length before the bulk of the computation is done in the transformer @xcite @xcite . none of these methods claim faster processing over a standard subword-level model, however, so there would still be a trade-off between accuracy and speed when applying these methods, albeit to a lesser extent. ## conclusion subword-level models are currently the dominant paradigm for machine translation. however, this work showed that character models could produce competitive or even superior results in many circumstances. first, character models obtain an overall better translation quality on trained language pairs. moreover, we highlighted how the gain in translation quality from using character models is particularly marked when fine-tuning data is scarce. finally, we showed how character models have superior cross-lingual transferability, especially with languages seen in pretraining, or those that are similar to a language seen in pretraining. following the results of our analyses, we can attribute this superior quality to a character model's ability to implicitly account for the appropriate input granularity given the context, translating at times in a word-by-word fashion, or characterby-character when needed. this ultimately results in better accuracy when translating orthographically similar and rare words. the quality increase is however not without a trade-off. indeed, character models are at least 4 times slower in training and inference, making them suboptimal for many real-world situations. we posit that further work into speeding up generation models @xcite @xcite , as well as more thorough evaluations of neural metrics for character models, would greatly benefit this area of research. nevertheless, we can conclude that in less time-sensitive, or low-resource settings, character-level translations are worth the wait. mitchell stern, noam shazeer, and jakob uszkoreit. 2018. blockwise parallel decoding for deep autoregressive models. in advances in neural information processing systems, volume 31. curran associates, inc. yi tay, vinh q. tran, sebastian ruder, jai gupta, hyung won chung, dara bahri, zhen qin, simon baumgartner, cong yu, and donald metzler. 2022. charformer: fast character transformers via gradient-based subword tokenization. in proceedings of the tenth international conference on learning representations (iclr), online. jörg tiedemann and preslav nakov. 2013. analyzing the use of character-level translation with sparse and noisy datasets. in proceedings of the international conference recent advances in natural language processing ranlp 2013, pages 676-684, hissar, bulgaria. incoma ltd. shoumen, bulgaria. elena voita, rico sennrich, and ivan titov. 2021a. analyzing the source and target contributions to predictions in neural machine translation. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1126-1140, online. association for computational linguistics. https://doi.org/10.18653/v1 /2021.acl-long.91 elena voita, rico sennrich, and ivan titov. 2021b. language modeling, lexical translation, reordering: the training process of nmt through the lens of classical smt. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 8478-8491, online and punta cana, dominican republic. association for computational linguistics. https://doi.org/10 .18653/v1/2021.emnlp-main.667 linting xue, aditya barua, noah constant, rami al-rfou, sharan narang, mihir kale, adam roberts, and colin raffel. 2022. byt5: towards a token-free future with pretrained byte-to-byte models. transactions of the association for computational linguistics, 10:291-306. https://doi.org/10.1162 /tacl a 00461 linting xue, noah constant, adam roberts, mihir kale, rami al-rfou, aditya siddhant, aditya barua, and colin raffel. 2021. mt5: a massively multilingual pre-trained text-to-text transformer. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 483-498, online. association for computational linguistics. https://doi.org/10.18653/v1 /2021.naacl-main.41 lili yu, dániel simig, colin flaherty, armen aghajanyan, luke zettlemoyer, and mike lewis. 2023. megabyte: predicting millionbyte sequences with multiscale transformers. arxiv preprint arxiv:2305.07185.