A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models (2023)

Abstract: Recent works on tokenizer-free multilingual pretrained models show promising results in improving cross-lingual transfer and reducing engineering overhead compared to subword-based alternatives. However, previous work mainly focuses on reporting accuracy on a limited set of tasks and data settings, placing less emphasis on other important factors when tuning and deploying the models in practice, such as memory usage, inference speed, and finetuning data efficiency. We attempt to fill this gap by performing a comprehensive empirical comparison of multilingual tokenizer-free and subword-based models considering the various dimensions. Surprisingly, we find that subword-based models might still be the most practical choice in many settings, achieving better performance for lower inference latency and memory usage. Based on these results, we encourage future work in tokenizer-free methods to consider these factors when designing and evaluating new models.

## introduction several recent results @xcite have excited the research community with the possibility of "tokenizer-free" models, character-level and byte-level models, as an alternative to more traditional subword-based models. tokenizer-free models are especially appealing to practitioners as they can eschew the two-step processing pipeline of subword segmentation and reduce the corresponding difficulties in cross-lingual transfer @xcite @xcite or domain adaptation @xcite due to inconsistent subword units. however, upon several attempts to apply tokenizer-free methods, our analysis reveals several practical difficulties in applying these methods. this paper is a chronicle of some of the concerns we uncovered; we highlight some challenges with applying these models and propose best practices for future results reporting in this area. specifically, we perform experiments finetuning pretrained multilingual models, evaluating them with respect to finetuning data efficiency, inference time, and memory consumption. based on these multiple dimensions, we come to the somewhat surprising conclusion that subword-based models, in particular mbert @xcite , might still be the most practical choice in most settings, as they perform best while maintaining a relatively low inference cost. ## tokenizer-free multilingual models while multilingual pretrained models @xcite @xcite have led to impressive performance improvements for low-resource languages through cross-lingual transfer, the standard word representation method in these models relies on subword segmentation @xcite . in multilingual settings, subword tokenization can be sub-optimal as supporting hundreds of languages with various scripts and vocabularies causes segmentation mismatch between languages and over-segmentation in the lower-resourced languages @xcite . to alleviate this problem, recent works propose removing the subword segmentation step by using characters or bytes as lexical units @xcite . in particular, these "tokenizer-free" methods have been applied to both encoder-only and encoder-decoder models. tab. 1 presents an overview of the different tokenizerfree multilingual models with comparable subword models. next, we briefly describe the two tokenizer-free models we consider in this work. canine @xcite is a character-level encoder suggested as an alternative to mbert @xcite . canine operates on raw characters and is pretrained using the masked language modeling objective. to compensate for the computational efficiency loss due to increased sequence length, canine uses convolutions to downsample the sequence before passing the representations to the transformer layers. the two weight variants of canine (canine-s, canine-c) have the same architecture but slightly different pretraining objectives using either subwords or characters at the last layer. as both variants performed similarly in our experiments and @xcite , we only include canine-s for the main discussion, leaving canine-c results in § b.3. byt5 @xcite is an encoder-decoder transformer model similar to the mt5 @xcite model. byt5 operates on the raw utf-8 bytes of the input without any downsampling, leading to a longer sequence length while having a much smaller vocabulary size than mt5. both byt5 and mt5 are pretrained on the mc4 corpus 2 using the span reconstruction objective proposed by @xcite . to keep the parameter count fixed between mt5 and byt5, byt5 allocates the parameters saved from the embedding layer to additional encoder layers. although adding more depth to the encoder is a reasonable design choice, our results in § 4 show that byt5 suffers from a much higher inference cost due to the deeper encoder, especially when input/output sequence lengths are longer. ## experimental settings we conduct a multi-dimensional evaluation focusing on two aspects: finetuning data efficiency ( § 4.1) and inference cost ( § 4.2) to provide a better understanding of the practical applicability of tokenizer-free models. we finetune and evaluate 2 https://www.tensorflow.org/datasets/catalog/ c4#c4multilingual two subword-based models (mbert, mt5) and two tokenizer-free models (canine, byt5), as mbert-canine and mt5-byt5 are directly comparable counterparts in terms of their pretraining corpus as shown in tab. 1. for the t5 models, we consider only the small models of both mt5 and byt5 as the focus of our work is in the practical implication of using multilingual pretrained models at relatively resource-constrained settings. specifically, we finetune the models on three multilingual natural language understanding tasks adopted from the xtreme benchmark @xcite . the three tasks we choose cover various input, output formats -sequence-level classification (xnli), token-level classification (ner), and extractive question answering (tydi qa-goldp). ner named entity recognition (ner) is a structured prediction task, where the model predicts a tag (location, person, organization) in iob2 format for each token in the input sentence. we use the wikiann dataset @xcite and select 20 out of 282 languages for multilingual training based on linguistic diversity and the language availability in the other two tasks we consider. tydi qa-goldp the typologically diverse question answering @xcite dataset is an extractive qa benchmark in 11 languages. while the original dataset includes two "primary" tasks (selectp, minspan), the secondary goldp task is the most widely adopted as it is compatible with other squad-style qa tasks @xcite . for this reason, we mainly compare models on tydi qa-goldp ), all in-language samples (single), or the entire multilingual dataset (multi). the exact numbers can be found in the appendix (tab. 2). and discuss primary task results briefly through our replication experiment of @xcite . ## related work large-scale nlp models have achieved remarkable performance in various natural language tasks, with the recent chatgpt demonstrating near humanlevel language understanding capabilities. while achieving impressive results in standard benchmark settings, the applicability of these models have remained limited mainly due to practical considerations including their high energy consumption and environmental impact @xcite . both the nlp and computer vision communities have proposed evaluating models based on practical metrics, such as training/inference efficiency @xcite @xcite , energy usage @xcite , robustness @xcite @xcite , and expected performance @xcite . similarly, a recent study by @xcite suggests a comprehensive evaluation suite for generative nlp models, including measures of robustness, fairness, and efficiency. our multi-dimensional evaluation is an attempt to expand these evaluation protocols to multilingual settings and examine the trade-offs of various tokenization schemes. ## conclusion in this paper, we present a multi-dimensional evaluation of tokenizer-free multilingual models focusing on their efficiency against finetuning dataset size and inference cost. based on our experiments, we find that mbert might still be the most cost-effective choice for many tasks, and show that the efficiency trade-offs of model design choices (tokenization, decoder availability) depend heavily on the task's length statistics. despite our findings, tokenizer-free models still have a significant advantage in reducing engineering effort and potentially increasing robustness to noisy data. we believe more work should be done in developing efficient tokenizer-free models, and encourage the community to consider these criteria of practical applicability when developing and evaluating tokenizer-free pretrained models.