BPE -Dropout: Simple and Effective Subword Regularization (2020)

Abstract: Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 2.3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization.

## introduction using subword segmentation has become de-facto standard in neural machine translation @xcite . byte pair encoding (bpe) @xcite is the dominant approach to subword segmentation. it keeps the common words intact while splitting the rare and unknown ones into a sequence of subword units. this potentially allows a model to make use of morphology, word composition and transliteration. bpe effectively deals with an openvocabulary problem and is widely used due to its simplicity. there is, however, a drawback of bpe in its deterministic nature: it splits words into unique subword sequences, which means that for each word a model observes only one segmentation. thus, a model is likely not to reach its full potential in exploiting morphology, learning the compositionality of words and being robust to segmentation errors. moreover, as we will show further, subwords into which rare words are segmented end up poorly understood. a natural way to handle this problem is to enable multiple segmentation candidates. this was initially proposed by @xcite as a subword regularization -a regularization method, which is implemented as an on-the-fly data sampling and is not specific to nmt architecture. since standard bpe produces single segmentation, to realize this regularization the author had to propose a new subword segmentation, different from bpe. however, the introduced approach is rather complicated: it requires training a separate segmentation unigram language model, using em and viterbi algorithms, and forbids using conventional bpe. in contrast, we show that bpe itself incorporates the ability to produce multiple segmentations of the same word. bpe builds a vocabulary of subwords and a merge table, which specifies which subwords have to be merged into a bigger subword, as well as the priority of the merges. during segmentation, words are first split into sequences of characters, then the learned merge operations are applied to merge the characters into larger, known symbols, till no merge can be done (figure 1(a) ). we introduce bpe-dropout -a subword regularization method based on and compatible with conventional bpe. it uses a vocabulary and a table); merges performed at each iteration are shown in green, dropped -in red. merge table built by bpe, but at each merge step, some merges are randomly dropped. this results in different segmentations for the same word (figure 1(b)). our method requires no segmentation training in addition to bpe and uses standard bpe at test time, therefore is simple. bpe-dropout is superior compared to both bpe and kudo (2018) on a wide range of translation tasks, therefore is effective. our key contributions are as follows: ## background in this section, we briefly describe bpe and the concept of subword regularization. we assume that our task is machine translation, where a model needs to predict the target sentence y given the source sentence x, but the methods we describe are not task-specific. ## our approach: bpe-dropout we show that to realize subword regularization it is not necessary to reject bpe since multiple segmentation candidates can be generated within the bpe framework. we introduce bpe-dropout -a method which exploits the innate ability of bpe to be stochastic. it alters the segmentation procedure while keeping the original bpe merge table. during segmentation, at each merge step some merges are randomly dropped with the probability p. this procedure is described in algorithm 1. algorithm 1: bpe-dropout current split ← characters from input word; do merges ← all possible merges 1 of tokens from current split; for merge from merges do / * the only difference from bpe * / remove merge from merges with the probability p; end if merges is not empty then merge ← select the merge with the highest priority from merges; apply merge to current split; end while merges is not empty; return current split; if p is set to 0, the segmentation is equivalent to the standard bpe; if p is set to 1, the segmentation splits words into distinct characters. the values between 0 and 1 can be used to control the segmentation granularity. we use @xmath0.1) in training time to expose a model to different segmentations and @xmath1, which means that at inference time we use the original bpe. we discuss the choice of the value of p in section 5. when some merges are randomly forbidden during segmentation, words end up segmented in different subwords; see for example figure 1(b) . we hypothesize that exposing a model to different 1 in case of multiple occurrences of the same merge in a word (for example, m-e-r-g-e-r has two occurrences of the merge (e, r)), we decide independently for each occurrence whether to drop it or not. ## analysis in this section, we analyze qualitative differences between models trained with bpe and bpedropout. we find, that ## related work closest to our work in motivation is the work by @xcite , who introduced the subword regularization framework multiple segmentation candidates and a new segmentation algorithm. other segmentation algorithms include @xcite , @xcite , chitnis and denero (2015), @xcite , @xcite , @xcite . regularization techniques are widely used for training deep neural networks. among regularizations applied to a network weights the most popular are dropout @xcite and l 2 regularization. data augmentation techniques in natural language processing include dropping tokens at random positions or swapping tokens at close positions @xcite @xcite , replacing tokens at random positions with a placeholder token @xcite , replacing tokens at random positions with a token sampled from some distribution (e.g., based on token frequency or a language model) @xcite @xcite . while bpe-dropout can be thought of as a regularization, our motivation is not to make a model robust by injecting noise. by exposing a model to different segmentations, we want to teach it to better understand the composition of words as well as subwords, and make it more flexible in the choice of segmentation during inference. several works study how translation quality depends on a level of granularity of a segmentation @xcite @xcite . @xcite show that trained long enough character-level models tend to have better quality, but it comes with the increase of computational cost for both training and inference. @xcite find that, given flexibility in choosing segmentation level, the model prefers to operate on (almost) character level. @xcite explore the effect of bpe vocabulary size and find that it is better to use small vocabulary for low-resource setting and large vocabulary for a high-resource setting. following these observations, in our experiments we use different vocabulary size depending on a dataset size to ensure the strongest baselines. ## conclusions we introduce bpe-dropout -simple and effective subword regularization, which operates within the standard bpe framework. the only difference from bpe is how a word is segmented during model training: bpe-dropout randomly drops some merges from the bpe merge table, which results in different segmentations for the same word. models trained with bpe-dropout (1) outperform bpe and the previous subword regularization on a wide range of translation tasks, (2) have better quality of learned embeddings, (3) are more robust to noisy input. future research directions include adaptive dropout rates for different merges and an in-depth analysis of other pathologies in learned token embeddings for different segmentations.