What is the best recipe for character-level encoder-only modelling? (2023)

Abstract: This paper aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. Many such modelling architectures and methods to train those architectures have been proposed, but it is currently unclear what the relative contributions of the architecture vs. the pretraining objective are to final model performance. We explore the design space of such models, comparing architectural innovations (Clark et al., 2022, Jaegle et al., 2022, Tay et al., 2021) and a variety of different pretraining objectives on a suite of evaluation tasks with a fixed training procedure in order to find the currently optimal way to build and train character-level BERT-like models. We find that our best performing character-level model exceeds the performance of a token-based model trained with the same settings on the same data, suggesting that character-level models are ready for more widespread adoption. Unfortunately, the best method to train character-level models still relies on a subword-level tokeniser during pretraining, and final model performance is highly dependent on tokeniser quality. We believe our results demonstrate the readiness of character-level models for multilingual language representation, and encourage NLP practitioners to try them as drop-in replacements for token-based models.

## introduction the first stage of almost all nlp modelling pipelines is to convert input text strings into a sequence of symbols that the model can ingest. this step, called tokenisation, can be highly non-trivial and introduces significant theoretical and practical complexities to both training and using these models. one particular issue for massively multilingual models is that many languages have to compete for space given a fixed vocabulary size, which limits the effective vocabulary size per language: as an illustration, the wordpiece tokeniser that multilin-gual bert uses tokenises 'hello' as two tokens: 'hell' and '##o'. 1 we are therefore interested in alternative approaches which use lightweight tokenisation schemes (in particular character-level segmentation) coupled with powerful neural-network based composition functions to build language models (see section 5.1 for a discussion of the benefits of character-level modelling). in this paper, we aim to determine the best way to build such models, focussing on models which output vector representations for each input character. however, as the field of pretrained characterlevel modelling is relatively new, comparisons are complicated by the fact that recently proposed methods use different model architectures, pretrain on different data using different training objectives, and evaluate on different downstream tasks. with so many variables changing simultaneously, it is difficult to disentangle the effect of each individual choice in the modelling pipeline, and therefore decide on an overall best model configuration. to answer this question, we tested many model architectures and pretraining objectives from recent literature on a unified set of evaluation tasks, with the same training procedure. we identify one particular configuration that shows the best performance across all of our downstream evaluation tasks, namely a combination of the charformer downsampling model @xcite , and canine upsampling model and pretraining procedure @xcite . we dub this configuration bort, for bidirectional orthographic representation technique. this model even outperforms a bert baseline on all tasks we consider, while being moderately slower to pretrain ( §4.1). one sticky point we discovered is that the best modelling configuration we found above relies crucially on a tokeniser during pretraining. we investigate alternative objectives that do not require a tokeniser, and find that these objectives result in worse-performing models. in addition, we also investigate the impact of the tokeniser used to pretrain the model, and find that the quality of the tokeniser (measured by vocabulary size) has a big impact on the final model downstream task performance, even though the tokeniser is not used at all during evaluation. this results in the unfortunate situation that users of such models have a hidden dependency on the tokeniser used to train the model; hence, users may be using models out of domain without any explicit feedback (such as worse tokeniser compression rates), causing difficult-todetect performance regressions. taken together, we believe our results show that character-level representation models are ready to supplant subword-level models as a default choice for converting text into features. however, these models still require extensive supervision from tokenisers, and we believe that the next frontier of research in character-level modelling is finding ways to once and for all eliminate tokenisation as a key step in the nlp pipeline. ## the ingredients to make a character-level encoder model in this section, we aim to give an overview of all the components necessary to make a performant and efficient encoder-only model which operates on characters and outputs contextualised character representations. working with characters rather than subword tokens brings many challenges, which have been solved in different ways in prior literature; we compare the selected methods in our experiments. in the following section, words in bold correspond to one cell in ## experiments we train all our models on the same multilingual wikipedia dump as mbert, with the same exponentially weighted language sampling strategy. our baseline model architecture is bert-base with 110m parameters; all other models are comparable in size. we train each model using 32 tpuv3 chips for 250k steps with total batch size 3072. models took between 3 and 4 days to complete training. we found the batch size parameter crucial for final model performance: using a smaller batch size degraded final model performance, while characterlevel model performance was unstable at a larger batch size. for exact pretraining hyperparameters and downstream task evaluation procedures, please see appendices a and b. unless otherwise stated, the hyperparameters are constant across all experiments; each experiment aims to examine the influence of a specific choice of variable. we evaluate model checkpoints on a rolling basis during training on all our evaluation tasks, and select the model checkpoint which performs the best on tydi-qa. ## conclusion in this paper, we examined how best to train a character-level encoder model, and identified a recipe that produces models exceeding the performance of token-based models at a comparable compute cost, suggesting that the time of general purpose character-level modelling has arrived.