Self-Vocabularizing Training for Neural Machine Translation (2025)

Abstract: Past vocabulary learning techniques identify relevant vocabulary before training, relying on statistical and entropy-based assumptions that largely neglect the role of model training.Empirically, we observe that trained translation models are induced to use a byte-pair encoding (BPE) vocabulary subset distinct from the original BPE vocabulary, leading to performance improvements when retrained with the induced vocabulary.In this paper, we analyze this discrepancy in neural machine translation by examining vocabulary and entropy shifts during self-training—where each iteration generates a labeled dataset by pairing source sentences with the model’s predictions to define a new vocabulary.Building on these insights, we propose *self-vocabularizing training*, an iterative method that self-selects a smaller, more optimal vocabulary, yielding up to a 1.49 BLEU improvement.Moreover, we find that deeper model architectures lead to both an increase in unique token usage and a 6–8% reduction in vocabulary size.

## introduction vocabulary construction, also known as vocabularization, is essential for many natural language processing tasks that involve neural networks, including neural machine translation (mt), as highlighted in various studies @xcite @xcite @xcite . however, past vocabulary learning techniques rely on corpus statistics such as entropy @xcite or frequency counts @xcite , without considering contextual information or the model's ability to represent it. despite the success of vocabularization in improving mt model efficiency @xcite , we observe a discrepancy between the original bytepair encoding (bpe) vocabulary @xcite vocabulary induced from pseudo-labeled data (v 1 ) (see figure 1 ). this discrepancy is surprising, as it suggests that mt models implicitly learn a pseudo-"optimal" vocabulary (v 1 ) that is substantially different from the original vocabulary (v 0 ) and is also smaller in size. for instance, on the iwslt14 de-en dataset, |v 1 | is approximately 20% smaller than |v 0 |. moreover, mt models retrained with the pseudo vocabulary v 1 outperform those trained on the original vocabulary set. this suggests a need to re-examine the assumptions underlying vocabulary learning techniques such as byte-pair encoding and the marginal utility of vocabularization @xcite , as existing methods may overlook modeldata interactions, leaving key optimization factors unaccounted for. in this paper, we aim to understand this discrepancy in neural machine translation models by analyzing shifts in vocabulary and entropy during self-training. to this end, we conduct experiments on two language tasks, comparing the vocabulary sets learned from the original training data and the pseudo-labeled data. our results suggest limited overlap between the two vocabularies and that pseudo data induces a more optimal vocabulary, enabling further improvements. furthermore, our findings indicate that the decoder has a limited impact on this vocabulary shift, whereas encoderbased interactions play a crucial role in entropy reduction. this suggests that future vocabulary induction methods should focus more on the crossattention module. finally, our study has implications for defining an optimal vocabulary set in language generation. this preliminary study also introduces a simple yet effective technique: iterative self-training to self-select a more optimal vocabulary set for performance gains. in summary, this paper makes the following contributions: (1) we identify a discrepancy between the optimal and pseudo-labeled vocabulary derived from mt models. (2) we analyze shifts in vocabulary and entropy during self-training. (3) we propose a simple approach to obtain a competitive vocabulary set and introduce a self-vocabularizing training algorithm that improves performance. ## iterative self-vocabularization current vocabularization techniques adopt two contrasting perspectives: (1) focusing on frequency or entropy statistics to avoid the computational cost of trial training @xcite , which often neglects important parameters and interactions in the process. (2) obtaining a more optimal vocabulary set through training @xcite , but at a higher computational cost. this work combines both perspectives by adopting an entropybased vocabularization approach while utilizing self-training. in self-training, a base model m t is trained on the dataset to generate predictions for input sequences, which are then used to update the next iteration of the base model m t+1 . this process is repeated iteratively with the supervised loss l from labeled instances @xcite , where x and y are the source and target texts, respectively: measuring vocabulary shifts subword-based approaches like byte-pair encoding are widely used and have demonstrated strong empirical performance @xcite @xcite @xcite @xcite @xcite . these methods construct vocabulary by selecting high-probability subword units. following @xcite , we define the vocabulary shift as the negative change in entropy normalized by vocabulary size: ## experimental settings for our experiments, we used the iwslt14 german-english parallel corpus for both germanto-english (de-en) and english-to-german (en-de) translation tasks. we preprocessed the data using moses @xcite and applied byte-pair encoding (bpe) @xcite to construct the vocabulary set. we evaluated model performance using bleu scores on the test set, comparing against the baseline and other comparable models. for selftraining, we ran each iteration until performance converged. in each iteration i, the model was trained from scratch in the self-training step (st-i) for analysis purposes. results were averaged over three initialization runs with different θ * . ## conclusions and findings in this paper, we investigated the discrepancy between the "optimal" vocabulary set identified prior to training a translation model and the vocabulary actually used by the trained model. we found that the trained model diverged from the original bpe vocabulary and that a single iteration of selftraining was sufficient to generate a competitive vocabulary set. additionally, we examined the relationship between the self-vocabularizing process and the encoder-decoder architecture, demonstrating that deeper models favor the selection of rarer tokens while reducing vocabulary size, whereas decoders have a lesser influence on vocabularization.