How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in J apanese (2023)

Abstract: This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case study. The tokenizer for such languages often consists of a morphological analyzer and a subword tokenizer, requiring us to conduct a comprehensive study of all possible pairs. However, previous studies lack this comprehensiveness. We therefore train extensive sets of tokenizers, build a PLM using each, and measure the downstream performance on a wide range of tasks. Our results demonstrate that each downstream task has a different optimal morphological analyzer, and that it is better to use Byte-Pair-Encoding or Unigram rather than WordPiece as a subword tokenizer, regardless of the type of task.

## introduction tokenization is the first key procedure in current natural language processing when inputting a target sentence to a pretrained language model (plm). it generally splits an input sequence into subword units, where a subword is a fraction of a word. previous efforts have proposed several subwordtokenization algorithms (hereafter, subword tokenizers), such as byte-pair-encoding (bpe) @xcite , wordpiece @xcite , and unigram @xcite , and different plms use different subword tokenizers. 1 it is widely acknowledged that tokenization affects the downstrem performance of plms @xcite @xcite @xcite . the majority of the previous studies have focused on languages with explicit word boundaries, such as english, while research on scriptio con- "_" and "/" denote a space and a subword boundary, respectively. figure 1 : typical tokenization procedures in both scriptio continua languages and english tinua languages, or languages without word boundaries (like japanese, chinese, and thai), is still understudied. the tokenization process in scriptio continua languages traditionally involves morphological analysis, which splits the input text into morphemes (semantic units similar to words in english) using the dictionary designed by human experts (see step 1 in figure 1 for an example). in this case, a tokenizer for a plm consists of a morphological analyzer and a subword tokenizer. to investigate the impact of tokenization in this scenario, we need to perform a comprehensive study on several sets of the available pairs, which is lacking in the previous work @xcite @xcite . in this paper, we investigate the effect of tokenizers on the downstream performance of plms in scriptio continua languages, focusing on japanese as a case study. we train an extensive collection of tokenizers consisting of known morphological analyzer and subword tokenizer pairs, use them to pretrain and fine-tune bert models, and measure their performance on a variety of downstream tasks. on the basis of the experimental results, we address the following three research questions. we first try to answer if we should use a morphological analyzer 2 in a scriptio continua language (japanese) (rq1). rq2 and rq3 each examine whether different morphological analyzers/subword tokenizers perform differently on a downstream task. contributions 1) we test a comprehensive set of known morphological analyzer and subword tokenizer pairs and use various downstream tasks to clarify the effect of tokenizers on the downstream performance of japanese plms. 2) accordingly, we find the followings: in this section, we explain the morphological analyzers and subword tokenizers used in this paper. ## experimental setup 7 tokenizers we compared a total of 12 tokenizers (four morphological analyzers and three subword tokenizers), as introduced in §2. we also considered three additional tokenizers not using morphological analyzers. we trained all tokenizers with the vocabulary size of 30k utilizing 10m sentences randomly extracted from japanese wikipedia. ## results and analysis this section addresses the three rqs raised in §1. rq1: should we use a morphological analyzer? in addition to the above, we observe that word-piece + nothing produced by far the worst results in all tasks due to the poor tokenization. wordpiece processes a sequence word by word and treats a sequence without a blank as a single word. if it fails to tokenize a particular word, it tokenizes the "whole" as a single [unk] token. without a morphological analyzer, the length of a word becomes abnormally long, making wordpiece more likely to produce an [unk] token. this means that the majority of an input text will be converted into [unk] tokens, thus losing almost all of the content in the text. in fact, the average sequence length and ratio of [unk] per sample in pretraining were 1.15 ± 3.28 and 99.8 ± 4.9%, respectively. these caused unstable pretraining (see appendix d). compared with other tasks, nothing in ner showed a considerable performance degradation with a maximum difference of 22.2 (juman++ vs. nothing in bpe). in ner, annotations are wordlevel and tend to align well with morphemes. since tokenizers with morphological analyzers split a morpheme into subword tokens, they can produce more linguistically motivated subword segmentation than nothing, thus giving them an advantage. rq2: do different morphological analyzers perform differently on downstream tasks? looking at the statistical test results for rq2 in for tasks with statistical significance, we further ran the steel-dwass test @xcite to see which morphological analyzer had a significant performance difference from the others ( rq3: do different subword tokenizers perform differently on downstream tasks? from the statistical test results for rq3 in we next analyze and discuss which differences in subword tokenizers produced downstream performance differences. first, we look at the difference in the vocabulary of subword tokenizers. we plot the relationship between vocabulary similarity and performance difference between two different subword tokenizers in figure 2 . the vocabulary similarity of two different subword tokenizers is computed as and unigram use a more sophisticated approach (as explained in §2.2). this algorithmic difference might also contribute to the performance difference between different subword tokenizers. ## conclusion to investigate the effect of tokenizers on the downstream performance of plms in a scriptio continua language (japanese), we compared extensive sets of tokenizers by evaluating them on a wide range of downstream tasks and addressed the three rqs in §1. future work will examine how to automatically select the optimal tokenizer pair for a given task.