Aligning Sizes of Intermediate Layers by L o RA Adapter for Knowledge Distillation (2025)

Abstract: Intermediate Layer Distillation (ILD) is a variant of Knowledge Distillation (KD), a method for compressing neural networks.ILD requires mapping to align the intermediate layer sizes of the teacher and student models to compute the loss function in training, while this mapping is not used during inference.This inconsistency may reduce the effectiveness of learning in intermediate layers.In this study, we propose LoRAILD, which uses LoRA adapters to eliminate the inconsistency.However, our experimental results show that LoRAILD does not outperform existing methods.Furthermore, contrary to previous studies, we observe that conventional ILD does not outperform vanilla KD.Our analysis of the distilled models’ intermediate layers suggests that ILD does not improve language models’ performance.

## introduction the llm's performance is rapidly improving on various natural language processing (nlp) tasks at the cost of the huge parameter size, resulting in enormous computational costs. therefore, reducing the parameter size while retaining the model's performance is an important research topic. knowledge distillation (kd) @xcite is one of the model compression methods. kd employs two models: a teacher model and a student model. the teacher model is already trained for a specific task. the teacher's output serves as soft labels that guide the student model that mimics the teacher's behavior. while kd usually uses the output of the teacher, intermediate layer distillation (ild) @xcite uses the information of intermediate layers as well; ild has been claimed to be superior to the vanilla kd in previous studies @xcite @xcite . in this study, we introduce loraild, which is designed to improve the conventional ild by employing the lora @xcite adapter in aligning sizes of intermediate layers between the teacher and the student. we evaluate the performance of loraild through empirical comparisons with conventional kd baselines. in general, we find that our loraild does not outperform the conventional ild baselines, and even the conventional ild is not necessarily superior to the vanilla kd, at least in our experimental settings. ## analysis 5.1 analysis method to examine how the intermediate layers of the models embed features in rail-kd and loraild, we conduct a cluster analysis of features used in loss calculation and intermediate layer outputs. for both the teacher and the student of loraild, rail-kd, and w/o ild (λ 3 = 0), we obtain h * we evaluate the clusters by calculating the adjusted rand index (ari) @xcite against gold clusters that are constructed according to the gold labels. the higher ari suggests the student acquires better representation for solving the task through the training. a higher ari for the teacher means its intermediate layers provide more useful information to train the student. ## conclusion our experimental result ( the rail-kd and other baselines used in our experiment employ different base models, datasets, and hyperparameters from their original ones, so our experiment does not completely replicate their original settings. we acknowledge that our teacher models are different from the previous studies. this is because the teacher models used in the previous study are not publicly available. also, the architecture of the teacher model we used differs from that of the previous studies because of the addition of the lora adapter. as for the data set, as described in section 4.1, our dataset split is different. 90% of the glue train set was used as our train set, the remaining 10% of the glue train set as a validation set, and the original validation set as a test set. due to the different conditions described above, we had to perform a hyperparameter search to find the optimal values for our settings.