Distributional Properties of Subword Regularization (2024)

Abstract: Subword regularization, used widely in NLP, improves model performance by reducing the dependency on exact tokenizations, augmenting the training corpus, and exposing the model to more unique contexts during training. BPE and MaxMatch, two popular subword tokenization schemes, have stochastic dropout regularization variants. However, there has not been an analysis of the distributions formed by them.We show that these stochastic variants are heavily biased towards a small set of tokenizations per word. If the benefits of subword regularization are as mentioned, we hypothesize that biasedness artificially limits the effectiveness of these schemes. Thus, we propose an algorithm to uniformly sample tokenizations that we use as a drop-in replacement for the stochastic aspects of existing tokenizers, and find that it improves machine translation quality.

## introduction tokenization is the first stage in almost all natural language processing pipelines, where raw text is transformed into a format that is understood by the model. modern neural models use subword tokenization, which represents text as a sequence of subword units drawn from a subword vocabulary (e.g., @xmath0. popular subword tokenization schemes are bpe @xcite , maxmatch/wordpiece @xcite , and unigramlm @xcite . unintentionally, the downstream models are thus not conditioned on the raw text, but rather the exact tokenization of the text. during training, subword regularization (where static tokenizations are replaced with sampled tokenizations) is often used to break the dependency on the exact tokenization. it also serves as data augmentation, and improves performance in a variety of downstream tasks. 0 our code is available at github.com/mcognetta/ distributional-properties-of-subword-regularization. bpe-dropout @xmath1.1 to ken ization 97.77% to ke n ization 1.89% to k en ization 0.25% to ken iz ation 0.04% t oken ization 0.03% to k en iz ation 0.01% to ke n iz ation 0.01% to ken i z @xmath2.01% maxmatch-dropout @xmath3.3 to ken ization 34.29% t oken ization 14.66% to ke n ization 10.48% to ken iz ation 7.21% t oke n ization 4.39% to k en ization 3.15% t oken iz ation 3.05% to ke n iz ation ## motivation though stochastic tokenization is known to improve model quality, it remains unclear which tokenization distribution is the best. bpe-and maxmatch-dropout, which induce unlearned distributions (probabilities that are not chosen by a learning algorithm), are a natural way of injecting randomness into the underlying tokenization algorithm post hoc. however, empirically, we see that they both induce heavily biased distributions, and hypothesize that an unbiased stochastic tokenizer would be universally better. this hypothesis is based on three explanations for subword regularization's effectiveness: 1) regularization subword regularization regularizes the model by breaking the dependency on a single, canonical tokenization. as shown in example 1, bpe-and maxmatch-dropout allocate most of their probability mass to only a few tokenizations for a given input. a tokenizer that uniformly samples from the distribution will expose the model to a greater variety of unique tokenizations of the same input text during training. ## subword tokenization subword tokenizers are typically deterministic in that the same character sequence will result in the same tokenized output sequence. stochastic variants were developed to allow for sampling tokenizations, which has been shown to improve model quality and robustness in a variety of nlp tasks. we briefly introduce three common subword tokenization schemes and their stochastic variants, which all share the same formalization. formalization. deterministic tokenization maps words @xmath4. an important part of tokenization is that it is lossless-a tokenization of an input can be inverted to recover the original word. for example, t(tokenization) = to ken ization but importantly t -1 (to ken ization) = tokenization. in contrast, stochastic tokenization is not a oneto-one mapping, but rather a probability distribution function t w for each word w. this assigns each tokenization w a probability t w ( w) ∈ [0, 1]. continuing example 1, t w (to ken ization) = 0.98 and t w (to ke n ization) = 0.02. during application of the tokenizer, the specific tokenization of w is sampled from the distribution t w . bpe and dropout. bpe forms a vocabulary by iteratively merging the most frequently cooccurring pair of tokens in the corpus @xcite , see appendix b, algorithm 4. this forms a vocabulary v and an ordered list of merges µ. during inference, the merges are applied greedily to the input (algorithm 1). until no more merges are available, the highest priority available merge is found and applied to the sequence. to implement dropout, a probability p is introduced, and the highlighted statement randomly removes candidate merges during each iteration @xcite . ## experiments we use english↔german, english↔romanian, and english↔french as our translation tasks. for each language pair, we train a baseline bpe and maxmatch tokenizer with the same vocabulary size and use them to build dropout and uniform sampler variants so that the vocabulary between a baseline tokenizer and its stochastic variants is exactly the same and only the tokenization distribution is different. we include a unigramlm tokenizer with the same vocabulary size as a learned-distribution baseline.we use the same underlying transformer model (appendix a) for each language pair, and only change the embedding and decoding layers, according to the choice of tokenizer. we compare tokenizer efficiency (via tokenization-scorer), bleu @xcite , chrf @xcite , and comet da-22 @xcite by averaging the results of three experimental runs per model. we use @xmath5.1 for bpe-dropout, @xmath6.3 for maxmatch-dropout, and α = 0.3 for unigramlm. these values were taken from the literature where they have been shown to perform well @xcite @xcite . 4 for our smallest dataset, en↔de, we experiment with the use of dropout with both the sourceand target-side tokenizers. however, for the largest, en↔fr, we use dropout only on the source side, in line with other work which has shown targetside dropout to not be helpful (and sometimes even harmful) in high-resource settings @xcite . for en↔ro, which is in between the size of the other two, we experiment with both settings. for uniform sampling, we use @xmath7.1 and 0.25, which were chosen as an estimate of the frequency that a non-canonical tokenization of word in bpe-and maxmatch-dropout was sampled, respectively. thus, we should expect uniform samplers to have roughly the same amount of noncanonically-tokenized-words in a corpus as bpeand maxmatch-dropout, so the salient difference is the variety of tokenizations. a subset of the results are shown in there is only one metric (@xmath8, bpe, chrf) where a uniform sampling model is not the best. however, in that same case, the uniform sampler improved upon the bpe-dropout model by 0.8 bleu, which is nearly as much as the bpe-dropout improved upon the bpe baseline. in addition, the +0.61 increase in comet da-22 corresponds to a 82% agreement accuracy with humans @xcite . in the @xmath9, uniform sampling models were the best across all metrics and underlying tokenizers. further, uniform sampling consistently outperforms un-igramlm both in terms of raw translation quality metrics and improvement over the deterministic baseline. in line with other research, the effects of subword tokenization lessen as the dataset size grows @xcite . still, the observed improvements extend to our full experimental results (appendix c) and support our hypothesis that an unbiased tokenizer should generally outperform biased dropout tokenizers. ## conclusion we investigate the distributions induced by bpeand maxmatch-dropout, two popular subword regularization schemes. we hypothesize and show that bpe-and maxmatch-dropout are suboptimal in that they form heavily biased distributions. we introduce a uniform sampler tokenizer, which guarantees uniform distributions and consistently outperforms bpe-and maxmatch-dropout on machine translation tasks. future work. uniform sampling is uniform at the word level, but past research suggests that uniformity at the global unigram level is desired @xcite . therefore, algorithms could be designed to directly optimize global uniformity. further investigations should reconcile how both uniform sampling and unigramlm improve performance despite their opposing motivations (higher/lower entropy).