When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation (2021)

Abstract: Subword segmentation algorithms have been a de facto choice when building neural machine translation systems. However, most of them need to learn a segmentation model based on some heuristics, which may produce sub-optimal segmentation. This can be problematic in some scenarios when the target language has rich morphological changes or there is not enough data for learning compact composition rules. Translating at fully character level has the potential to alleviate the issue, but empirical performances of character-based models has not been fully explored. In this paper, we present an in-depth comparison between character-based and subword-based NMT systems under three settings: translating to typologically diverse languages, training with low resource, and adapting to unseen domains. Experiment results show strong competitiveness of character-based models. Further analyses show that compared to subword-based models, character-based models are better at handling morphological phenomena, generating rare and unknown words, and more suitable for transferring to unseen domains.

## introduction neural machine translation (nmt) has achieved great success in recent years. modern nmt systems typically operate on subword level, using segmentation algorithms such as byte pair encoding (bpe) @xcite or morfessor @xcite . compared to word-level models, subword segmentation helps overcome the out-ofvocabulary (oov) problem and make better use of morphological information in the surface form. despite their empirical effectiveness, subword algorithms may produce improper segmentation due to their data-dependent nature. nmt models are typically robust to such errors when trained on large corpora or the target language is regular in morphological changes, like french or german. however, the problem will arise when such conditions are not met, i.e. there is not enough data for learning compact composition rules or the target language is morphologically rich and complex. an alternative segmentation choice is to use fully character-level (char) models @xcite @xcite @xcite , which has the potential to alleviate above issues. char does not need to learn any segmentation rules and keeps all available information in the surface form, avoiding the risk of information loss due to improper segmentation. what is more, the main pain point of char that it takes too long to train is less obvious in above settings since there is not as much data as in the rich resource setting. however, there has not been a comprehensive study in these settings. in this paper, we conduct a systematic comparison between char and other subword algorithms, e.g. bpe and morfessor. experiments show strong competitiveness of char under three settings: translating to typologically diverse languages (section 2), training with low resource (section 3), and adapting to distant domains (section 4). further analyses show that compared to subword algorithms, the benefits of char mainly come from better capture of the morphological phenomena, better generation of rare and unknown words, and better translation of domain-specific words. ## translation across typologically diverse languages human languages are known to exhibit diverse morphological phenomena, which could serve as a principle to classify languages into different morphological categories, such as fusional, agglutinative, introflexive and isolating. while previous ## translation with low resource subword algorithms help alleviate the oov problem. however, most of them are based on heuristics and may produce wrong segmentation. while this problem is not so evident when there is enough data to learn robust composition rules, in low-resource setting it could be a different story and their effectiveness should be examined. while for char, pure character sequences can directly provide all the information to the model for learning the composition rules. therefore a prudent choice of segmentation should be studied in this setting. ## translation across distant domains domain robustness @xcite , which refers to models' generalization ability on unseen domains, is important for nmt applications. however, subword algorithms need to learn segmentation rules from a given corpus, which may be domain-specific. when applied to a new domain, they may improperly segment target-domain specific words, hurting the domain robustness. in contrast, char does not suffer from the issue. in this section, we investigate how different segmentation algorithms affect nmt models' domain robustness. ## related work character-level neural machine translation has received growing attention in recent years. @xcite first propose a fully character-level nmt model based on recurrent encoder-decoder architecture and convolutional layers, which shows a promising results. @xcite propose to incorporate convolution layers in the more advanced transformer architecture and show their model can learn more robust character-level alignments. however, translating at character level may incur significant computational overhead. therefore, later works on character-level nmt @xcite ) mainly focus on reducing computation cost of them. @xcite show that by employing source sequence compression techniques, the quality and efficiency of character-based models can be properly balanced. @xcite share the same idea as @xcite but build their models using transformer architecture. our work differs from theirs in that we aim to analyze the performance of existing models instead of exploring novel architectures. there are also several researches on comparison between char and other subword algorithms @xcite . @xcite compare character-based models and subword-based models in terms of representation quality, and find that representation learned by the former are more suitable for modeling morphology, and more robust to noisy input. @xcite investigate the performance of different segmentation algorithms when using transformer architecture, and find that character-based models can achieve better performance when translating noisy text or text from a different domain. our finds are consistent with them, yet we conduct a more large-scale and in-depth analysis by covering language pairs from more language families and explaining where the advantage of character-based models comes from. ## conclusion we conduct a comprehensive study and show advantages of char over subword algorithms in three settings: translating to typologically diverse languages, translating with low resource, and adapting to distant domains. note that although we have tried our best to take as much language pairs as possible into consideration, there are certainly a lot of languages remaining uncovered in this paper. however, we believe our experimental results can serve as an evidence of character-based nmt models' strong competitiveness. we hope more attention will be drawn to them, including exploring their more benefits and reducing the possibly higher computation cost in practice.