M ax M atch-Dropout: Subword Regularization for W ord P iece (2022)

Abstract: We present a subword regularization method for WordPiece, which uses a maximum matching algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a search using the maximum matching algorithm. It realizes finetuning with subword regularization for popular pretrained language models such as BERT-base. The experimental results demonstrate that MaxMatch-Dropout improves the performance of text classification and machine translation tasks as well as other subword regularization methods. Moreover, we provide a comparative analysis of subword regularization methods: subword regularization with SentencePiece (Unigram), BPE-Dropout, and MaxMatch-Dropout.

## introduction subword regularization @xcite is a wellknown technique for improving the performance of nlp systems, whereby a model is trained with various tokenizations that are sampled for each training epoch. this approach provides data augmentation and model robustness against tokenization differences. kudo (2018) first introduced subword regularization using a unigram language model that was included in their tokenization tool, namely sen-tencepiece @xcite , and reported its effectiveness on machine translation tasks. @xcite proposed a subword regularization method for byte pair encoding (bpe) known as bpe-dropout and demonstrated the superiority of their method over that using the unigram language model in machine translation tasks. moreover, subword regularization contributes to the performance improvement of text classification tasks @xcite . as subword regularization is implemented as a modification of a tokenizer, each method is specialized to a particular tokenizer type. for example, the original subword regularization @xcite is specialized to a tokenizer that uses the unigram language model and bpe-dropout is specialized to the bpe-based tokenizer. however, these existing subword regularization tools cannot be directly applied to the other common tokenizers such as wordpiece @xcite . wordpiece is a tokenizer that is based on the maximum matching algorithm. it is used as the default tokenizer for the popular pretrained language model bert @xcite . although the widely used bert models (e.g., bert-base) can improve the performance of various nlp tasks, subword regularization cannot be used for the finetuning of the model because no subword regularization method exists for wordpiece. the use of subword regularization for the finetuning of pretrained models with wordpiece may result in a further performance improvement. in this paper, we present a simple modification of wordpiece for the use of subword regularization. the proposed method, which is known as maxmatch-dropout, randomly drops words in a vocabulary during the tokenization process. that ## maximum matching a simple modification to the maximum matching algorithm is implemented so that maxmatch-dropout can realize subword regularization. prior to explaining the modification, we briefly review the maximum matching on which the proposed method is based 1 . given a vocabulary and a single word, the maximum matching searches the longest subword in the vocabulary and greedily tokenizes the word into a sequence of subwords from beginning to end. for example, let the vocabulary be composed of {a, b, c, d, abc, bcd}. the tokenizer with the maximum matching divides a word "abcd" into "abc, d" 2 . as the maximum matching searches subwords from the beginning of the word, this word is not tokenized as "a, bcd." when an input word includes an unknown character, such as "abce," the tokenizer replaces this word with a special token, " @xcite ." this tokenization process is usually implemented using a trie. the detailed tokenization process using the maximum matching for this example with the trie (figure 4 ) is described in appendix a. 1 song et al. (2021) explains the efficient implementation of the maximum matching in detail. 2 we do not use special tokens for a subword that begins in the middle of a word (e.g., "##") for simple explanation. ## proposed method: maxmatch-dropout the proposed method extends the maximum matching with an additional dropout process. this method randomly replaces accepting states into non-accepting states with dropped states. that is, accepting tokens are randomly skipped with a specified probability q, where q is a hyperparameter. figure 1 depicts the tokenization process of a word "word" with a vocabulary that includes {w, o, r, d, or, rd, word}. although the maximum matched subword beginning with the first character is "word" in the vocabulary, in this case, the state corresponding to "word" is dropped. thus, the latest accepted subword "w" is yielded and the next matching begins from the second character. finally, the tokenization process results in "w, or, d." this process is also outlined in algorithm 1 foot_0 . in the algorithm, w i,i+j denotes a subword beginning from the i-th character and ending with the (i + j -1)-th character in the word w, where |w| and |s| are the lengths of the input word and subword, respectively. moreover, ber(1 -q) denotes a bernoulli distribution that returns 1 with a probability of 1 -q. the tokenization process of maxmatch-dropout is detailed in the regularization strength can be tuned using the hyperparameter q. the proposed method is equivalent to the original maximum matching with @xmath0.0, and it tokenizes a word into characters with @xmath1.0 if all characters are included in the vocabulary. the official code is available at https: //github.com/tathi/maxmatch_ dropout. ## experiments we conducted experiments on text classification and machine translation tasks to validate the performance improvement provided by maxmatch-dropout. we used two tokenizers and subword regularization methods as a reference for both tasks: senten-cepiece (unigram) @xcite with subword regularization (sub. reg.) (kudo, 2018) and bpe @xcite with bpe-dropout @xcite . we employed wordpiece @xcite , which was implemented by huggingface @xcite , as a basic tokenizer for the proposed maxmatch-dropout 4 . we set the vocabulary size of each tokenizer to be equal to compare the three methods as fairly as possible. the vocabulary of each tokenizer included all characters that appeared in the training splits. we selected the hyperparameters for the subword regularization (e.g., q of maxmatch-dropout) according to the performance on the development splits. note that we could not fairly compare the performance of maxmatch-dropout to that of other subword regularization methods because they are based on different tokenizers and vocabularies. wordpiece was used as the baseline for maxmatch-dropout to investigate whether the method could successfully perform subword regularization and improve the performance similarly to other methods. ## conclusion we have introduced a subword regularization method for wordpiece, which is a common tokenizer for bert. the proposed method, maxmatch-dropout, modifies the tokenization process using the maximum matching to drop words in the vocabulary randomly. this simple modification can realize subword regularization for wordpiece. furthermore, the experimental results demonstrated that maxmatch-dropout can improve the performance of bert. maxmatch-dropout is also effective in the training of text classification tasks without bert and machine translation tasks, as well as existing subword regularization methods.