Subword-Delimited Downsampling for Better Character-Level Translation (2022)

Abstract: Subword-level models have been the dominant paradigm in NLP. However, character-level models have the benefit of seeing each character individually, providing the model with more detailed information that ultimately could lead to better models. Recent works have shown character-level models to be competitive with subword models, but costly in terms of time and computation. Character-level models with a downsampling component alleviate this, but at the cost of quality, particularly for machine translation. This work analyzes the problems of previous downsampling methods and introduces a novel downsampling method which is informed by subwords.This new downsampling method not only outperforms existing downsampling methods, showing that downsampling characters can be done without sacrificing quality, but also leads to promising performance compared to subword models for translation.

## introduction character-level models (henceforth character models) have recently sparked interest in their potential applicability across a wide range of nlp tasks. they promise a tokenization-free approach, while also potentially allowing the model to quickly recognize similarities between words based on their spelling. however, as of yet, character models have not stood up to the level of subword models, mainly due to their similar performance while being significantly slower and more expensive to train due to their longer input sequences @xcite . to alleviate the problem of training time, several methods have been proposed to initially downsample characters into shorter sequences, which are then fed into the encoder or decoder. for discriminative tasks, these can be applied without any loss in performance @xcite , however for generative tasks like nmt, the performance is either untested or lacking when compared to character models without downsampling @xcite . seeing as subword tokenization is essentially a form of downsampling and performs quite well, the idea of downsampling is not inherently flawed. however, attempts to downsample within the neural network have not achieved similar performance for translation. this begs the question, why are current neural downsampling methods underperforming for translation? in this work, we make three main contributions: we start by providing an overview of the prior work in character-level nlp in section 2. we then compare the various downsampling methods, noting the 3 main advantages of downsampling based on subwords in section 3. next, we cover the modifications necessary to do variable-length downsampling for translation in section 4. we follow this with experimental details (section 5), results (section 6), analysis (section 7), discussion (section 8), conclusion (section 9), and finally the limitations (section 10). ## related work character-level models have been of interest for several years, notably used for character-level translation prior to the advent of transformer models with some success @xcite . @xcite raises the issue that the length of the sequences require the models to potentially capture much longer range dependencies, and as such introduces a downsampling method. this method consists of convolutional layers followed by a sequence-length-wise max pooling. the convolutional layers serve to learn local patterns in the characters and the max pooling is intended to reduce the length of the input, alleviating the longrange dependency issue. reducing the length with a downsampling method such as max pooling can be thought of as a transformation from character tokens to pseudo-word tokens. more recently, character-level nlp has been investigated with the use of the transformer. the transformer, while better able to handle longer sequences than rnns, can similarly suffer when on the character-level due to the o(n 2 ) complexity of self-attention. nevertheless, byt5 @xcite , a multilingual unsupervised pretrained character-level model, has shown comparable results to its subword-level counterpart mt5, while demonstrating some beneficial properties such as robustness to character-level noise. it is however slower than subword models both in training and test time. the charformer @xcite reintroduces downsampling using a novel downsampling method, gbst, which uses a learned, weightedaverage of character n-grams for each downsampled token. this shows similar performance to byt5 while also being faster, however its performance on generative tasks such as nmt appears less promising. @xcite investigate the usefulness of charformer's gbst method for nmt, finding that using gbst decoder-side does not work out-of-the-box due to an information leak, and that even with a fix to the leak, it does not perform up to the level of the aforementioned convolutional downsampling method. similar to the charformer, characterbert @xcite shows that incorporating character information can be useful on encoder-only tasks. they use a cnn similar to @xcite 's, but downsample based on the length of the whole word, rather than at a fixed size. their results show better generalization than subword models on classification of medical data, despite it not seeing any medical data in pretraining. they attribute this to its more generalized internal vocabulary as a result of receiving characters as input. in the context of nmt, libovickỳ et al. ( 2021 ) attempt to answer why the current state-of-the-art models are not character models, to which the answer appears that their performance is not superior to subword models, and that downsampling methods sacrifice quality for efficiency. in doing so, libovickỳ et al. convert existing character models such as @xcite 's to the transformer architecture. with this, they propose a two-step decoding method, which adds an lstm layer that takes as input the hidden representation of the transformer decoder, concatenated with separately-learned character embeddings. the light-weight nature of the two-step decoder means little computation time is added. we show a tabular summary of the relevant previous work in ## exploring downsampling methods we first compare our novel subword-based downsampling method over other possible forms of downsampling. our choice of downsampling based on subwords is motivated by 3 factors: as a third method, buffered fixed-size downsampling ("buffered fixed"), which we introduce to better understand the importance of position and morpheme consistency. ## architecture we now explain the architecture used in our experiments. we build off of the previous work by using the cnn downsampling architecture followed by the transformer and using @xcite 's two-step decoding with an lstm for upsampling. this previous work was only applied to fixed-length downsampling and upsampling, however the aforementioned wdd and sdd methods require variable-length downsampling and upsampling. thus, we explain how this is accomplished in the next two subsections. figure 1 shows the architectures used in our translation experiments. foot_0 ## experimental setup our code is made available on github. 4 we experiment with translation, using the encoder-decoder, as well as two encoder-only tasks: nli and review classification. 5 while we focus mainly on improving translation, we also include these encoder-only tasks to test the importance of the choice of downsampling method for non-generative tasks. we compare several models, including the 4 downsampling methods (fixed, buffered fixed, wdd, and sdd) , as well as a subword-level model and a character-level model, 6 both of which use the standard transformer architecture, requiring no downsampling or upsampling module. for translation, we experiment with 3 language pairs: english-arabic, english-german, and english-turkish. we chose these language pairs as they exhibit different levels of linguistic similarity and morphological richness. we evaluate our models with bleu @xcite and comet @xcite . the full details of the datasets used for all tasks can be found in appendix a. to keep our research eco-friendly and to allow for faster iteration on our models, we train our models on smaller translation datasets, consisting of roughly 200 thousand sentence pairs per language pair, and roughly 500 thousand and 5 million sentences for nli and review classification, respectively. we discuss the potential for these models in the high-resource setting in section 8. for sdd, we change the value of the max_sentencepiece_length argument in sentencepiece to achieve an effective downsam-4 https://github.com/leukas/sdd 5 review classification is classifying product reviews from 1 to 5 stars based on the review title and content. our parameter setup for models, vocabulary, and training are noted in appendix b. ## results we start by comparing the 4 downsampling methods, discerning the importance of the positional, length, and morpheme consistency. we then compare our best-performing downsampling method to our two standard baselines: the subword-level and character-level models. third, we compare these models in their ability to generalize to outof-domain datasets. lastly, we look at the encoderonly tasks, comparing all methods thus far. ## embedding analysis the sole difference in the architecture of the encoder-only models is their embeddings, or the hidden representations prior to being fed into the transformer. as such, we extract these embeddings for words to analyze their differences. we compare the models in their word embedding similarity (i.e. cosine similarity) for pairs of words. we generate 4 test sets: the first two sets are meant to distinguish beneficial character-level similarity (e.g. "take" and "takes") from detrimental similarity (e.g. "pour" and "tour"). the latter two sets are meant to distinguish the opposite, when a lack of similarity may be appropriate (e.g. "bulb" and "faster") or inappropriate (e.g. "camp" and "tent"). we limit the vocabulary used in the test sets to words in the subword vocabulary that only require a single token. we also filter out tokens which have no synonyms, effectively also removing non-words and non-english words. lemmas and synonyms are gathered using wordnet @xcite . we split the test sets additionally into 3 subcategories: "seen", where both words in the pair have been seen during training, "half-seen", where only one word is seen in training, and "unseen", where neither is seen in training. 9 the sizes of each split are shown in as expected, the subword model has near 0 similarity for half-seen and unseen words since it has no mechanism for developing their embeddings during training. it does recognize grammatical seen words as similar, but it has a more muted response to seen words with similar spelling. in contrast, sdd has a strong response to both seen and unseen words, as the character models can develop embeddings for unseen words according to character-level patterns. we see that both models appear capable of distinguishing between grammatical pairs and pairs with similar spelling but not necessarily similar meaning, though only in the case where both words are seen for the subword model. there is a stark difference between the z-scores of the close spell and far spell sets for the sdd model. as for the far synonyms, there is a small difference for the subword model, and none for the sdd model. it is possible that these words are distinguished in later layers of the model. ## practicality discussion while this paper takes a more theoretical approach, using smaller, domain-specific datasets for training nmt, the practical usage is worth considering. given the prior work of byt5, charformer, and characterbert, the consensus appears that with large, pretrained models using character information, the performance on standard metrics is similar, and the outperformance is on out-of-domain data @xcite or data with characterlevel corrupted input @xcite . as such, we expect the same is true for sdd, however the upsampling module used (based on @xcite ) appears to be a limiting factor. we explain 9 for these experiments, we used separate models trained using the same initial vocabulary as t5-small, in order to get a larger number of single-token words to evaluate on. this is why there exist words in the vocabulary that are not seen during training. the performance of these models is similar to those reported in conversely, we expect such models to be useful in lower-resource settings. in such an scenarios, we show that the inclusion of character-level information improves performance beyond that of subword models (appendix d). translation encoder tasks iter epochs total iter epochs total char ×3.58 ×1.31 ×4.69 ×2.23 ×2.04 ×4.55 sdd ×2.61 ×1.28 ×3.34 ×1.34 ×1.36 ×1.82 in terms of training time, we report the training time ratios with respect to the subword model in one potential use of the sdd model that is yet unexplored comes in the form of adapting existing subword models to take character input. since pretraining is a costly endeavor, and since there are beneficial characteristics of character-level models such as out-of-domain generalization, it may be possible to improve a subword model by adapting it to use the sdd downsampling module rather than its own subword embeddings. previous work in adaptation has shown success in adapting models to different tasks @xcite and languages @xcite @xcite , so a similar approach may be useful here. ## conclusion previous work has casted doubt on the usefulness of character-level nmt models, due to their lack of improvement over subword models despite using more fine-grained information while being also slower. downsampling modules added to the character models have previously been proposed but have always come at the cost of accuracy. we show that it is possible to downsample without sacrificing accuracy, by downsampling based on the lengths of subwords. this novel downsampling outperforms the previous downsampling methods, as well as character and subword models on the majority of language pairs tested. there are several avenues for future research. while much work has been done on optimizing a vocabulary for a subword model, finding the optimal lengths for subword-delimited downsampling is still an open problem. the most promising may in fact be adapting any of the numerous pretrained subword models to use characters as input. overall, character-level models show promise that has yet to be fully realized.