Finding the Optimal Vocabulary Size for Neural Machine Translation (2020)

Abstract: We cast neural machine translation (NMT) as a classification task in an autoregressive setting and analyze the limitations of both classification and autoregression components. Classifiers are known to perform better with balanced class distributions during training. Since the Zipfian nature of languages causes imbalanced classes, we explore its effect on NMT. We analyze the effect of various vocabulary sizes on NMT performance on multiple languages with many data sizes, and reveal an explanation for why certain vocabulary sizes are better than others.

## introduction natural language processing (nlp) tasks such as sentiment analysis @xcite and spam detection are modeled as classification tasks, where instances are independently labeled. tasks such as part-of-speech tagging @xcite and named entity recognition @xcite are examples of structured classification tasks, where instance classification is decomposed into a sequence of per-token contextualized labels. we can similarly cast neural machine translation (nmt), an example of a natural language generation (nlg) task, as a form of structured classification, where an instance label (a translation) is generated as a sequence of contextualized labels, here by an autoregressor (see section 2). since the parameters of modern machine learning (ml) classification models are estimated from training data, whatever biases exist in the training data will affect model performance. among those biases, class imbalance is a topic of our interest. class imbalance is said to exist when one or more classes are not of approximately equal frequency in data. the effect of class imbalance has been extensively studied in several domains where classifiers are used (see section 6.3). with neural networks, the imbalanced learning problem is mostly targeted to computer vision tasks; nlp tasks are under-explored @xcite . word types in natural language models resemble a zipfian distribution, i.e. in any natural language corpus, we observe that a type's rank is roughly inversely proportional to its frequency. thus, a few types are extremely frequent, while most of the rest lie on the long tail of infrequency. zipfian distributions cause two problems in classifier-based nlg systems: the use of subwords, that is, decomposition of word types into pieces, such as the widely used byte pair encoding (bpe) @xcite addresses the open-ended vocabulary problem by ultimately allowing a word to be represented as a sequence of characters if necessary. bpe has a single hyperparameter named merge operations that governs the vocabulary size. the effect of this hyperparameter is not well understood. in practice, it is either chosen arbitrarily or via trial-and-error @xcite . regarding the problem of imbalanced classes, @xcite states that "the machine learning techniques that we rely on are actually very bad at inducing systems for which the crucial information is in rare events." however, to the best of our knowledge, this problem has not yet been directly addressed in the nlg setting. in this work, we attempt to find answers to these questions: 'what value of bpe vocabulary size is best for nmt?', and more crucially an explanation for 'why that value?'. as we will see, the answers and explanations for those are an immediate consequence of a broader question, namely 'what is the impact of zipfian imbalance on classifier-based nlg?' the contributions of this paper are as follows: we offer a simplified view of nmt architectures by re-envisioning them as two high-level components: a classifier and an autoregressor (section 2). we describe some of the desired settings for the classifier (section 2.1) and autoregressor (section 2.2) components. in section 2.3, we describe how vocabulary size choice relates to the desired settings for the two components. our experimental setup is described in section 3, followed by an analysis of results in section 4 that offers an explanation with evidence for why some vocabulary sizes are better than others. section 5 uncovers the impact of class imbalance, particularly frequency based discrimination on classes. 2 section 6 provides an overview of related work, and in section 7 we recommend a heuristic for choosing the bpe hyperparameter. ## classifier based nlg machine translation is commonly defined as the task of transforming sequences from the form @xmath0...x m to @xmath1...y n , where x is in source language x and y is in target language y . there are many variations of nmt architectures (section 6.1), however, all share the common objective of maximizing q n @xmath2, x 1:m ) for pairs (x 1:m , y 1:n ) sampled from a parallel dataset. nmt architectures are commonly viewed as encoderdecoder networks. we instead re-envision the nmt architecture as two higher level components: an autoregressor (r) and a token classifier (c), as shown in figure 1 . autoregressor r, @xcite being the most complex component of the nmt model, has many implementations based on various neu-2 in this work, 'type' and 'class' are used interchangeably. ral network architectures: recurrent neural networks (rnn) such as long short-term memory (lstm) and gated recurrent unit (gru), convolutional neural networks (cnn), and transformer (section 6.1). at time step t, r transforms the input context @xmath3, x 1:m into hidden state vector h @xmath4, x 1:m ). classifier c is the same across all architectures. it maps h t to a distribution p (y j |h t )8y j 2 v y , where v y is the vocabulary of y . in machine learning, input to classifiers such as c is generally described as features that are either hand-engineered or automatically extracted. in our high-level view of nmt architectures, r is a neural network that serves as an automatic feature extractor for c. ## experimental setup our nmt experiments use the base transformer model @xcite on four different target languages at various training data sizes, described in the following subsections. ## measuring classifier bias due to imbalance in a typical classification setting with imbalanced classes, the classifier learns an undesired bias based on frequencies. a balanced class distribution debiases in this regard, leading to improvement in the precision of frequent classes as well as recall of infrequent classes. however, bleu focuses only on the precision of classes; except for adding a global brevity penalty, it is ignorant of the poor recall of infrequent classes. therefore, the bleu scores shown in figures 3a, 3b and 4 capture only a part of the improvements and biases. in this section we perform a detailed analysis of the impact of class balancing by considering both precision and recall of classes. we accomplish this in two stages: first, we define a method to measure the bias of the model for classes based on their frequencies. second, we track the bias in relation to vocabulary size and class imbalance, and report de!en, as it has many data points. ## conclusion envisioning nmt as a token classifier with an autoregressor helps in analysing its weaknesses. our analysis provides an explanation of why text generation using bpe vocabulary is more effective compared to word and character vocabularies, and why certain bpe hyperparameters are better than others. we show that the number of bpe merges is not an arbitrary hyperparameter, and that it can be tuned to address the class imbalance and sequence length problems. our recommendation for transformer nmt is to use the largest possible bpe vocabulary such that at least 95% of classes have 100 or more examples in training. even though certain bpe vocabulary sizes indirectly reduce the class imbalance, they do not completely eliminate it. the class distributions after applying bpe contain sufficient imbalance for inducing the frequency based bias, especially affecting the recall of rare classes. hence more effort in the future is needed to directly address the zipfian imbalance.