On the Effectiveness of Quasi Character-Level Models for Machine Translation (2022)

Abstract: Neural Machine Translation (NMT) models often use subword-level vocabularies to deal with rare or unknown words. Although some studies have shown the effectiveness of purely character-based models, these approaches have resulted in highly expensive models in computational terms. In this work, we explore the benefits of quasi-character-level models for very low-resource languages and their ability to mitigate the effects of the catastrophic forgetting problem. First, we conduct an empirical study on the efficacy of these models, as a function of the vocabulary and training set size, for a range of languages, domains, and architectures. Next, we study the ability of these models to mitigate the effects of catastrophic forgetting in machine translation. Our work suggests that quasi-character-level models have practically the same generalization capabilities as character-based models but at lower computational costs. Furthermore, they appear to help achieve greater consistency between domains than standard subword-level models, although the catastrophic forgetting problem is not mitigated.

## introduction neural machine translation (nmt) has become the dominant paradigm in the field of machine translation due to the impressive results obtained with the encoder-decoder architectures @xcite @xcite and the approaches proposed to tackle the open vocabulary problem such as subword-based models with byte-fallback or, more recently, token-free models. however, despite these advances, low-resource languages are still problematic as there are many languages that are spoken but not written on the internet (e.g., tigrinya, sotho, tsonga, etc.), and therefore, parallel text mining techniques are either not effective or not applicable at all. in these cases, it is common to use character-based models, since multiple authors have shown that these models usually perform better than (standard) subword-or word-based models in very low-resource settings. motivated by these ideas, we decided to study whether quasi-character-based vocabularies (defined as a subword-based vocabulary that is one or two orders of magnitude smaller than a standard subword-based vocabulary), had the same advantages as models with character-based vocabularies for low-resource languages, but with much lower computational costs, due to the exponential decrease in the average number of tokens per sentence when merging highly frequent character pairs. furthermore, given that the effects of the catastrophic forgetting problem are strongly related to the vocabulary of the model, we decided to study if these quasi-character-level vocabularies had the potential to mitigate them, since these vocabularies are closer to a universaldomain vocabulary (e.g., bytes or chars) than a word-or (standard) subword-based vocabulary. the contributions of this paper are twofold: ## related work character-based models have been widely studied in the field of natural language processing (nlp) to deal with the open vocabulary problem. @xcite proposed one of the first character-based models, who treated source and target sentences as a string of letters. similarly, @xcite viewed translation as a single transduction between character strings in the source. however, these results were unsatisfactory, as their models generally performed worse than their word-based counterparts. to overcome these problems, many authors have proposed strategies based on hybrid models @xcite , which mainly translated at the word level except when a rare or unknown word was encountered; subword-based model @xcite @xcite , which allow to efficiently represent a word as a sequence of subwords; and more recently, token-free models @xcite , which operate directly on raw text. despite these improvements, character-based models are still interesting for low-resource languages since multiple authors have shown their benefits over other approaches. for example, @xcite showed that character-level models have their greatest advantage when data sizes are small; @xcite showed that reducing the vocabulary size leads to improvements for low-resource nmt models. similarly, by studying the zipfian nature of languages in nmt, other authors have reached similar conclusions. @xcite characterized the long-tailed phenomena in nmt, and @xcite proved that each dataset has an optimal vocabulary size. although this optimal vocabulary size has been traditionally found by trial and error, very recently, @xcite has proposed a new technique to explore automatic vocabularization without trial and error. however, despite its impressive results, this method still requires a non-trivial amount of time foot_1 . hence, heuristics will remain an effective solution to the vocabulary-size problem. in this work, we focused our efforts on preserving the advantages of character-based models but at much lower computational costs. in this line, other authors have introduced new ideas, such as @xcite , who used convolutional and max-pool layers to reduce the length of the character-level representations; @xcite showed that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences; @xcite proposed an approach to learning input and output segmentations for nmt, which favors character-level approaches; and more recently, @xcite published a survey about tokenization and the open-vocabulary problem, where they concluded that it is likely that there will never be a silver bullet solution for all applications. this work briefly studies the generalization capabilities of quasi-character-level models for different neural architectures. many authors have extensively studied the limitations of existing tokenizations and neural architectures for text processing tasks. for instance, @xcite showed a state-of-the-art cnn architecture for text processing that operated directly at the character level; @xcite showed that the effectiveness of transformer under low-resource conditions is highly dependent on the hyper-parameter settings; @xcite presented a fast character transformer via gradient-based subword tokenization. finally, this work ends with a brief discussion on the ability of quasi-character-level models to mitigate the effects of the catastrophic forgetting problem in nmt. as far as we know, this is the first work to address this problem from this perspective, since most of the works that we know of are based on regularization @xcite , dynamic architectures @xcite or complementary learning systems (cls) @xcite . ## conclusion in this work, we have studied the effectiveness of quasi-character-level models in terms of performance and computational efficiency relative to purely character-based models and standard subword-level models. furthermore, we have studied the generalization of quasi-character-level vocabularies and their ability to address the problem of catastrophic forgetting. our studies reveal that quasi-character-level models offer practically the same generalization capabilities as character-level models, but at much lower computational costs. furthermore, these models outperformed both the standard subword-based and character-based models in low-resource environments, regardless of language, domain, and neural architecture. finally, we have shown that even though quasi-character-level models do not appear to mitigate the effects of the catastrophic forgetting problem, they achieved better cross-domain consistencies, which could lead to substantial improvements if specific regularization techniques are applied to deal with the catastrophic forgetting problem.