BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation (2022)

Abstract: BPE tokenization merges characters into longer tokens by finding frequently occurring contiguous patterns within the word boundary. An intuitive relaxation would be to extend a BPE vocabulary with multi-word expressions (MWEs): bigrams ( in\_a ), trigrams ( out\_of\_the ), and skip-grams ( he . his ). In the context of Neural Machine Translation (NMT), we replace the least frequent subword/whole-word tokens with the most frequent MWEs. We find that these modifications to BPE end up hurting the model, resulting in a net drop of BLEU and chrF scores across two language pairs. We observe that naively extending BPE beyond word boundaries results in incoherent tokens which are themselves better represented as individual words. Moreover, we find that Pointwise Mutual Information (PMI) instead of frequency finds better MWEs (e.g., New\_York , Statue\_of\_Liberty , neither . nor ) which consistently improves translation performance.We release all code at https://github.com/pegasus-lynx/mwe-bpe .

## introduction subword tokenization algorithms like byte pair encoding (bpe) @xcite group together frequently occurring patterns, such as -ing or -ly, into individual tokens. the success of subword tokenization points to the benefit in modeling longer patterns, even though any given text can be represented simply as a sequence of characters. this paper stretches the motivation further by allowing bpe to cross word boundaries. in the context of nmt, we find that the straightforward way to find mwes by bpe (sorted by frequency) hurts performance whereas sorting by pmi scores improves scores. we hypothesize and discuss a reason for these observations and provide further recommendations on using mwes with bpe. n-gram tokens have been used in traditional nlp for a long time and with much success. for example ( raw he lives in new york . tok he_ lives_ in_ new_york_ ._ raw i love the statue of liberty! tok i_ love_ the_ statue_of_liberty_ !_ raw she lost her bag . tok she_ • her_ @xmath0._ this paper experiments with two ways to expand bpe with mwes for the task of nmt. concretely, we promise the following contributions: ## methods mwes have been commonly used in traditional nlp but rarely in the age of transformers and subword vocabularies. here we describe two kinds of ways to add mwes to a bpe vocabulary. ## datasets we use the iit bombay hindi-english parallel corpus v3.0 @xcite , tokenized using indicnlplibrary @xcite and moses tokenizer @xcite respectively. the train : dev : test splits have 1.6m : 0.5k : 2.3k sentences respectively. for german-english, the datasets are retreived from the news translation task of wmt2019 @xcite . the train : dev : test splits have 4.5m : 3k : 2k sentences respectively. while we use the originally mentioned training set for our main results in our baseline vocabulary is bpe with 8k subword tokens for hi-en and 16k for de-en. each of our methods maintains the same vocabulary size, replacing the least frequently occurring subwords with corresponding n-grams or skip-grams. we show representative mwes learned from corpora in our nmt model is a 6 layer transformer encoder-decoder @xcite that has 8 attention heads, 512 hidden vector units, and a feed forward intermediate size of 2048, with gelu activation. we use label smoothing at 0.1, and a dropout rate of 0.1. we use the adam optimizer with a controlled learning rate that warms up for 16k steps followed by a decay rate recommended for training transformer models. we trim longer sequences to a maximum of 512 tokens after bpe. each model is trained from scratch, and the hyperparameters (per language pair) are chosen by grid search to optimize the baseline validation bleu. we train all models for up to 100k steps (batch @xmath3. the number of tokens replaced in the original bpe vocabulary with a corresponding mwe ordered by pmi, is also a hyperparameter optimized by grid search between 1.25% to 10% of the vocabulary size (hi-en models performing best when 1.25% tokens were replaced and de-en models performing best at 2.5% for bigrams/trigrams and 5% for skipgrams). we make sure to not replace any rare base characters like q or @. for ablations (section 5.2) with limited compute budget, we train hi-en models for up to 200k steps.we apply a patience of 10 validations, each 1000 update steps apart. to decode, we average the best 3 checkpoints, and use a beam size of 4 with length penalty of 0.6. we use nlcodec and rtg libraries @xcite and contribute our extensions to them as well. ## results and discussion moreover, since the methods of extracting mwes is purely emprirical and is language agnostic, the results and observations can be extended for different language pairs. we now attempt to reason why bpe fails beyond word boundaries in its vanilla form, and why switching to pmi solves the problem. we also study where does it help the most to add mwes. unless noted otherwise, the analysis is reported on the hi-en dataset. ## related work attempts at merging nmt with mwes typically include pairing up the network with a phrase based smt system @xcite @xcite and hierarchical phrases are expressive enough to cover discontinuous mwes @xcite . @xcite add manually annotated mwes aligned across the source and target language (en-it). however, this might not work for low resource languages, hence we extract mwes automatically with pmi. they count discontinuous mwes, one of our main contributions, among future work. multi-word tokens have a proven track record in nlp. skip-gram tokens, for instance, have already been used in phrase-based machine translation @xcite @xcite to tackle cases where certain phrases in a source language (duonianlai in chinese) are better represented as skip-grams in a target language (over the last • years in english) @xcite . our work revisits these ideas and adapts them to a transformer-based nlp model relying on subword segmentation. there also exists prior work on defining, counting, and evaluating kskip-n-grams @xcite @xcite , although unrelated to the task of nmt. finally, readers interested in other applications of extracting mwes via pmi scores may refer to @xcite where similar techniques are used to efficiently mask tokens while pretraining bert @xcite . ## conclusion this paper systematically studies the impact of extending a bpe vocabulary with multi-word expressions for neural machine translation. our results point to the vast unexplored scope of different granularities of tokenization that can be exploited by nlp systems. notably, our methods extend to not only longer contiguous tokens like n-grams but also skip-grams, which have been relatively unexplored with transformer-based nlp. in future work, we intend to compare our pmibased methods to human-annotated mwes as well as to recent workarounds to interfering tokenization schemes such as subword regularization or bpe dropout @xcite ). we also wish to extend experiments to nlp tasks beyond nmt, and the scope of our tokens to, say, variable-skip-grams which allow for any number of skips.