Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping (2025)

Abstract: Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families ( i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements.

## introduction knowledge distillation (kd) @xcite @xcite ; guo et al., 2025) has emerged as a promising methodological framework for enhancing the performance of compact models through knowledge transfer from larger, more powerful teacher models. nevertheless, conventional kd approaches, which aim to minimize the distribution difference between the logits of teachers and students, are still restricted by the requirement for tokenizer consistency between teachers and students. to address this limitation, cross-tokenizer kd (ctkd) @xcite has emerged as a critical research frontier. as illustrated in figure 1 , the core challenges of ctkd arise from two fundamental aspects: (1) divergent tokenization strategies induce sequence misalignment during text processing, and (2) vocabulary discrepancies create dimensional and semantic mismatches in output probability spaces. both the sequence and vocabulary level misalignments create significant barriers to effective knowledge transfer between heteroge-neous architectures. current approaches attempt to bridge these gaps through two primary strategies: to quantify the discrepancies in sentence tokenization and vocabulary among mainstream large language models (llms) equipped with different tokenizers, we conducted a comprehensive analysis of alignment rates 2 across five mainstream llms. the results demonstrate the wide range of sequence alignment rates (around 30%-90%) and vocabulary alignment rates (around 10%-90%), which collectively highlight the substantial room for improvement in cross-tokenizer knowledge transfer. to further enhance the accuracy of model output alignment, we introduce contextual dynamic mapping (cdm), a novel ctkd framework that introduces: building on this mapping framework, we futher prioritize top-k tokens based on their contextual significance, which effectively suppresses noise while enhancing reasonable token mapping rates further. the experiments are based on five encompassing open-source model series, and the training and evaluation contain various tasks, including instruction following, code generation, and math. the experimental results demonstrate that cdm exhibits consistent and substantial superiority over current mainstream cross-tokenizer distillation approaches among different tasks (e.g., for qwen2-0.5b model, cdm has the improvement of 4.27% on instruction 2 the detailed statistical process is shown at section 2 following tasks, 12.19% on code tasks and 3.34% on math tasks) and even surpasses the performance of same-tokenizer distillation in some settings. our main contributions are as follows: ## preliminary analysis to systematically investigate the fundamental challenges in ctkd, we analyze the alignment rates in the sequence and vocabulary levels separately. on the sequence level, we calculate the tokenization result of the two selected tokenizers. concretely, for a certain sentence, we statistically analyze the set of tokens overlap and average on the sample number as final results. the statistic is based on 3000 examples sampled from the training corpus of dolly-15k @xcite on the vocabulary level, we calculate the ratio of tokens that exactly matches the two selected tokenizers. the statistical results presented in figure 2 and figure 3 highlight two critical limitations: (1) vocabulary-level alignment rates, which reach as low as 9%, and (2) sequence-level alignment rates, with minimum values as low as 31%. these findings underscore the critical need for supplementary mapping mechanisms within existing methodologies. furthermore, the intersection of these two mismatch dimensions creates compounding effects, namely, sequence alignment errors amplify vocabulary mismatches through error propagation. this issue motivates our proposed contextual dynamic mapping framework, which mitigates alignment errors by utilizing contextual information. ## analysis our analysis experiments adopts phi3-mini-3.8b and qwen2-0.5b as a default configuration, mainly including quantitative measurements of sequencelevel and vocabulary-level alignment improvements (section 5.1), comparative analysis with sequence-level knowledge distillation (section 5.2), and systematic exploration of dual-teacher distillation paradigms (section 5.3). we introduce "average if" as a composite metric aggregating performance across five instruction-following tasks to streamline result interpretation. moreover, the supplementary analysis, including time cost analysis (section a.2), evaluation on general instructionfollowing testset (section a.3), ablation studies (section a.4), sensitivity analyses (section a.5), and comparative case studies of alignment outcomes (sections a.6-a.7), are comprehensively documented in the appendix. ## related work in the context of cross-tokenizer distillation, several methods have been proposed to align the probability distributions of models before performing distillation. this alignment typically involves both sequence and vocabulary dimensions. fu et al. (2023) aligns sequences through dynamic programming and aligns vocabularies through exact matching. to improve vocabulary alignment, the @xcite series introduced methods such as mined and statistical matching for fuzzy matching to supplement vocabulary alignment. despite these advancements, their effectiveness remains constrained by the prevalence of numerous mismatches. beyond alignment strategies based on text character similarity, @xcite proposed the use of optimal transport to quantify the distance between model logits. furthermore, @xcite refine this approach by optimizing the cost function at both sequence and vocabulary levels, effectively integrating both local and global information to improve overall performance. additionally, dskd @xcite introduces a dual alignment framework that simultaneously aligns hidden states and model logits. however, both uld and dskd methods suffer from ineffi-ciencies due to underutilization of the vocabulary. ## conclusion and future work in this work, we propose contextual dynamic mapping (cdm), a novel approach to cktd. cdm enhances the ctkd by improving the alignment of model outputs on both sequence and vocabulary levels through the use of online context information. specifically, the method incorporates entropy-based weights in the sequential alignment process and employs contextual top-k token pairs to dynamically map vocabulary probabilities. through extensive experiments across three groups of teacher-student configurations and three types of tasks (instruction-following, code generation and math), our method demonstrates significant advantages over existing approaches and shows further potential in dual-teacher scenarios. furthermore, statistical analyses and case studies are presented to demonstrate how the method enhances model alignment accuracy in both sequence and vocabulary. in future work, we plan to further scale the application of cdm by assessing its performance on more diverse training datasets and larger student and teacher models to enhance scalability. in particular, the dual-teacher distillation will be extended to multi-teacher settings for further observation.