Why don’t people use character-level machine translation? (2022)

Abstract: We present a literature and empirical survey that critically assesses the state of the art in character-level modeling for machine translation (MT). Despite evidence in the literature that character-level systems are comparable with subword systems, they are virtually never used in competitive setups in WMT competitions. We empirically show that even with recent modeling innovations in character-level natural language processing, character-level MT systems still struggle to match their subword-based counterparts. Character-level MT systems show neither better domain robustness, nor better morphological generalization, despite being often so motivated. However, we are able to show robustness towards source side noise and that translation quality does not degrade with increasing beam size at decoding time.

## introduction the progress in natural language processing (nlp) brought by deep learning is often narrated as removing assumptions about the input data and letting the models learn everything end-to-end. one of the assumptions about input data that seems to resist this trend is (at least partially) linguistically motivated segmentation of input data in machine translation (mt) and nlp in general. for nmt, several papers have claimed parity of character-based methods with subword models, highlighting advantageous features of such systems. very recent examples include @xcite ; @xcite ; @xcite . despite this, character-level methods are rarely used as strong baselines in research papers and shared task submissions, suggesting that character-level models might have drawbacks that are not sufficiently addressed in the literature. in this paper, we examine what the state of the art in character-level mt really is. we survey existing methods and conduct a meta-analysis of the input segmentation methods used in wmt shared task submissions. we then systematically compare the most recent character-processing architectures, some of them taken from general nlp research and used for the first time in mt. further, we propose an alternative two-step decoder architecture that unlike standard decoders does not suffer from a slow-down due to the length of character sequences. following the recent findings on mt decoding, we evaluate different decoding strategies in the character-level context. many previous studies on character-level mt drew their conclusions from experiments on rather small datasets and focused only on quantitatively assessed translation quality without further analysis. to compensate for this, we revisit and systematically evaluate the state-of-the-art approaches to character-level neural mt and identify their major strengths and weaknesses on large datasets. ## character-level neural mt character-level processing was hardly possible within the statistical mt paradigm that assumed the existence of phrases consisting of semantically rich tokens that roughly correspond to words. neural sequence-to-sequence models @xcite @xcite do not explicitly work with this assumption. in theory, they can learn to transform any sequence into any sequence. the original sequence-to-sequence models used word-based vocabularies of a limited size and which led to a relatively frequent occurrence of out-of-vocabulary tokens. a typical solution to that problem is subword segmentation @xcite , which keeps frequent tokens intact and splits less frequent ones into smaller units. modeling language on the character level is attractive because it can help overcome several problems of subword models. one-hot representations of words or subwords do not reflect systematic character-level relations between words, potentially harming morphologically rich languages. with subwords, minor typos on the source side lead to radically different input representations resulting in low robustness towards source-side noise @xcite . models using recurrent neural networks (rnns) showed early success with character-level segmentation on the decoder side @xcite . using character-level processing on the encoder side proved harder which was attributed to the features of the attention mechanism which can presumably benefit from semantically rich units (such as subwords) in the encoder. following this line of thinking, @xcite introduced 1d convolutions with max-pooling that pre-process the character sequence into a sequence of latent wordlike states. coupled with a character-level decoder, they claimed to match the state-of-the-art subwordbased models. even though this architecture works well on the character level, it does not generalize further to the byte level @xcite . hybrid approaches combining tokenization into words with the computation of character-based word representations were successfully used with rnns @xcite @xcite . later, @xcite showed that rnns perform on par with subword models without changing the model architecture if the models are sufficiently large. @xcite support this by showing that rnn models which learn segmentation jointly with the rest of the model are close to character-level. character-level modeling with transformers appears to be more difficult. @xcite used transparent attention @xcite to train deep character-level models and needed up to 32 layers to close the gap between the bpe and character models, which makes the model too large for practical use. @xcite narrowed the gap between subword and character modeling using curriculum learning by finetuning subword models to character-level. @xcite proposed adding a convolutional sub-layer in the transformer layers. at the cost of a 30% increase in parameter count, they managed to narrow the gap between subword-and character-based models by half. @xcite reused the convolutional preprocessing layer with constant-size segments of @xcite in a transformer model for translation into english. without changing the decoder, they reached comparable, but usually slightly worse, translation quality compared to bpe-based models. shaham and levy (2021a) revisited characterand byte-level mt on rather small iwslt datasets. their results show that character-level and bytelevel models are usually worse than bpe models, but byte-based models without embedding layers often outperform bpe-based models in the out-of-english direction. using similarly small datasets, @xcite claim that character-level modeling outperforms bpe when translating into fusional, agglutinative, and introflexive languages. nikolov et al. (2018) experimented with character-level models for romanized chinese. these models performed comparable to models using logographic signs, but significantly worse than models using subwords. @xcite argued that signs in logographic languages carry too much information and were able to improve the translation quality by segmenting chinese and japanese into sub-character units while keeping subword segmentation on the english side. little is known about other properties of character-level mt beyond the overall translation quality. @xcite prepared a set of contrastive english-german sentence pairs and tested them using shallow rnn-based models. they observed that character-based models transliterated better, but captured morphosyntactic agreement worse. @xcite evaluated transformer-based character-level models using morpheval and came to mixed conclusions. @xcite and libovický and fraser (2020) make claims about the noise robustness of the character-level models using synthetic noise. @xcite evaluated domain robustness by training models on small domain-specific datasets and evaluating them on unrelated domains, claim-ing the superiority of character-level models in this setup. on the other hand, @xcite evaluated the domain robustness in a more natural setup and did not observe higher robustness when evaluating general domain models on domain-specific tests compared to bpe. another consideration is longer training and inference times. character-level systems are significantly slower due to the increased sequence length. @xcite reported a 5.6-fold slowdown at training time and a 4.7-fold slowdown at inference time compared to subword models. recent research on character-level modeling goes beyond mt. pre-trained multilingual representations are a particularly active area. @xcite propose canine. the model shrinks character sequences into fewer hidden states (similar to @xcite . they use local self-attention and strided convolutions (instead of highway layers and max-pooling as in lee's work). their model is either trained using the masked-language-modeling objective @xcite with subword supervision, or in an encoder-decoder setup similar to @xcite . both methods reach a representation quality comparable to similar subword models. byt5 @xcite and charformer @xcite are based on the mt5 model @xcite which uses sequence-to-sequence denoising pre-training. whereas byt5 only uses byte sequences instead of subwords and differs in hyperparameters, charformer uses convolution and combines character blocks to obtain latent subword representations. these models mostly reach similar results to sub-word models, occasionally outperforming a few of them, in the case of charformer without a significant slowdown. ## wmt submissions the conference on machine translation (wmt) organizes annual shared tasks in various use cases of mt. the shared task submissions focus on translation quality rather than the novelty of presented ideas, as most other research papers do. therefore, we assume that, if character-level models were a fully-fledged alternative to subword models, at least some systems submitted to the shared tasks would use character-level models. we annotated recent system description papers with the input and output segmentation method they used. we focused on information about exper- among the 145 annotated system description papers, there were only two that used characterlevel segmentation. mahata et al. (2018) used a character-level model for finnish-to-english translation. this system, however, makes many suboptimal design choices and ended up as the last one in the manual evaluation. scherrer et al. (2019) experimented with character-level systems for similar language translation and observed that characters outperform other segmentations for spanish-portuguese translation, but not for czech-polish. @xcite experimented with different subword vocabulary sizes for english-inuktikut translation and reached the best results using a subword vocabulary of size 1k, which makes it close to the character level. most of the papers do not even mention character-level segmentation as a viable alternative they would like to pursue in future work (7% in 2018, 2% in 2019, none in 2020). character-level methods were more frequently used in wmt17 with rnn-based systems, especially for translation of finnish @xcite and less successfully for chinese @xcite and the automatic post-editing task @xcite . on the other hand, figure 1 shows that the research interest in character-level methods remains approximately the same, or may have slightly increased. for practical solutions in wmt systems, we clearly show that system designers in the wmt community have avoided character-level models. we speculate that the main reasons for not considering character-level modeling are its lower efficiency and the fact that the literature shows no clear improvement of translation quality. most of the submissions use back-translation (85%, 82%, and 94% in the respective years), often iterated several times (11%, 20%, 16%), which requires both training and inference on large datasets. with the approximately 5-fold slowdown, wmt-scale experiments on character models are not easily tractable. ## evaluated models we evaluate several transformer-based architectures for character-level mt. a major issue with character-level sequence processing is the sequence length and low information density compared to subword sequences. architectures for characterlevel sequence processing typically address this issue by locally processing and shrinking the sequences into latent word-like units. in our experiments, we explore several ways to do this. first, we directly use character embeddings as input to the transformer. second, following @xcite , we use the convolutional character processing layers proposed by @xcite . third, we replace the convolutions with local selfattention as proposed in the canine model @xcite . finally, we use the recently proposed charformer architecture @xcite . lee-style encoding. @xcite process the sequence of character embeddings with convolutions of different kernel sizes and number of output channels. in the original paper, this was followed by 4 highway layers @xcite . in our preliminary experiments, we observed that a too deep stack of highway layers leads to diminishing gradients, and we replaced the second two high-way layers with feedforward sublayers as used in the transformer architecture @xcite . canine. @xcite experiment with character-level pre-trained sentence representations. the character-processing architecture is in principle similar to @xcite but uses more modern building blocks. character embeddings are processed by a transformer layer with local selfattention which only allows the states to attend to states in their neighborhood. this is followed by downsampling using strided convolution. originally, canine used a local self-attention span as long as 128 characters. in the case of mt, this would usually span the entire sentence, so we use significantly shorter spans. charformer. unlike previous approaches, charformer @xcite does not apply a nonlinearity on the embeddings and gets latent subword representations by repeated averaging of character embeddings. first, it processes the sequence using a 1d convolution, so the states are aware of their mutual local positions in local neighborhoods. second, non-overlapping character n-grams of length up to n are represented by averages of the respective character embeddings. this means that for each character, there is a vector that represents the character as a member of n-grams of length 1 to n . in the third step, the character blocks are scored with a scoring function (a linear transformation), which can be interpreted as attention over the n different n-gram lengths. the attention scores are used to compute a weighted average over the n-gram representations. finally, the sequence is downsampled using mean-pooling with window size and stride size n (i.e., the maximum n-gram size). whereas lee-style encoding allows using lowdimensional character embeddings and keeps most parameters in the convolutional layers, canine and charformer need the character representation to have the same dimension as the following transformer layer stack. two-step decoding. the architectures mentioned above allow the transformer layers to operate more efficiently with a shorter and more information-dense sequence of states. however, while decoding, we need to generate the target character sequence in the original length, by outputting a block of characters in each decoding step. our preliminary experiments showed that generating blocks of characters non-autoregressively leads to incoherent output. therefore, we propose a twostep decoding architecture where the stack of transformer layers operating over the downsampled sequence is followed by a lightweight lstm autoregressive decoder (see figure 3 ). the input to the lstm decoder is a concatenation of the embedding of the previously generated character and a projection of the transformer decoder output state. at inference time, the lstm decoder generates a block of characters and inputs them to the character-level processing layer. the transformer decoder computes an output state that the lstm decoder uses to generate another character block. more details are in appendix a. modifying charformer for the two-step decoding would require a long padding at the beginning of the sequence causing the decoder to diverge. because of that, we use lee-style encoding on the decoder side when using charformer in the encoder. first, we conduct all our experiments on the small iwslt datasets. then we evaluate the most promising architectures on larger datasets. ## experiments on small data we implement the models using huggingface transformers @xcite . we take the ca-nine layer from huggingface transformers and use an independent implementation of charformer foot_0 . our source code is available on github. 2 hyperparameters and other experimental details can be found in appendix b. ## experiments on wmt data based on the results of the experiments with the iwslt data, we further experiment only with the lee-style encoder using a downsampling factor of 3 on the source side. additionally, we experiment with hybrid systems with a subword encoder and character decoder. we train translation systems of competitive quality on two high-resource language pairs, english-czech and english-german, and perform an extensive evaluation. ## conclusions in our extensive literature survey, we found evidence that character-level methods should reach comparative translation quality as subword methods, typically at the expense of much higher computation costs. we speculate that the computational cost is the reason why virtually none of the recent wmt systems used character-level methods or mentioned them as a reasonable alternative. recently, most innovations in character-level modeling were introduced in the context of pretrained representations. in our comparison of character processing architectures (two of them used for the first time in the context of mt), we showed that 1d convolutions followed by highway layers still deliver the best results for mt. character-level systems are still mostly worse than subword systems. moreover, the recent character-level architectures do not show advantages over vanilla character models, other than improved speed. to overcome efficiency issues, we proposed a two-step decoding architecture that matches the speed of subword models, however at the expense of a further drop in translation quality. furthermore, we found that conclusions of recent literature on decoding in mt do not generalize for character models. character models do not suffer from the beam search curse and decoding methods based on sampling perform poorly, here. evaluation on competitively large datasets showed that there is still a small quality gap between character and subword models. character models do not show better domain robustness, and only slightly better morphological generalization in german, although this is often mentioned as important motivation for character-level modeling. the only clear advantage of character models is high robustness towards source-side noise. in contrast to earlier work on character-level mt, which claimed that decoding is straightforward and which focused on the encoder part of the model, our conclusions are that lee-style encoding is comparable to subword encoders. even now, most modeling innovations focus on encoding. character-level decoding which is both accurate and efficient remains an open research question.