An Analysis of BPE Vocabulary Trimming in Neural Machine Translation (2024)

Abstract: We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a tokenization postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in model implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to consistently improve model performance, and is even prone to incurring heavy degradation.

## introduction subword tokenization is an important process in modern neural language modeling, as it enables models to represent any possible word over a known alphabet while keeping the vocabulary size small. one of the most common subword tokenization methods is byte-pair encoding (bpe; @xcite , a greedy, statistical subword tokenization method. bpe builds its vocabulary and tokenizes a corpus by iteratively replacing the most frequently co-occurring token pair with a single, merged token. an unfortunate side-effect of this process is the existence of "intermediate" subwords-subwords that appear during the process of forming longer subwords, but rarely appear as output tokens in the final sequence. vocabulary trimming is a tokenization postprocessing step where subwords that appear fewer than a prescribed number of times in a given corpus are replaced with their component subwords, with the intent of removing rare tokens for which the model cannot learn a robust representation @xcite . given a b-tokenized sequence t 1 , t 2 , . . . , t n , a trimmed bpe tokenizer produces a new sequence we perform a comprehensive study to understand the actual effect of vocabulary trimming on the performance of machine translation systems. in general, we find that vocabulary trimming has no consistent positive effect on model quality, and in many cases can substantially degrade it. ## experiments to determine the effect of vocabulary trimming, we use the iwslt14 @xmath0. for all experiments, we use the same underlying transformer-iwslt architecture from fairseq @xcite , and only vary the embedding and decoding layers of the model by changing the tokenizer's source and target vocabulary sizes, b s and b t (or b j for the joint-vocabulary setting), and source and target thresholds, t s and t t , respectively. for the jointvocabulary setting, a single tokenizer was formed by setting a vocabulary size and training the tokenizer on the concatenation of the source and target corpora. this baseline tokenizer was used to form separate source and target trimmed tokenizers. as seen in on the other hand, (10k, 10k) was found to be the worst performing split-vocabulary baseline. trimming this baseline increased bleu, but not enough to match the better performing baseline models. for another baseline, (8k, 8k), trimming did not consistently improve or degrade bleu. comet shows a slight positive trend in most settings. in all but one case, trimming with a threshold of (100, 100) lead to an improvement in over the baseline. curiously, in the 10k jointvocabulary setting, the trimmed models all have higher comet scores than the baseline, while all but one have lower bleu scores. we conclude that vocabulary trimming should be done with caution, as it does not consistently improve performance, can heavily degrade performance, and comes at the cost of longer sequence lengths. this conclusion is based on the results from the complete results and code to reproduce them will be made public in our forthcoming full article.