C 2 KD : Cross-layer and Cross-head Knowledge Distillation for Small Language Model-based Recommendation (2025)

Abstract: Sequential recommenders predict users’ next interactions based on historical behavior and are essential in modern recommendation systems. While Large Language Models (LLMs) show promise, their size and high inference costs limit deployment on resource-constrained devices. Small Language Models (SLMs) provide a more efficient alternative for edge devices, but bridging the recommendation performance gap between LLMs and SLMs remains challenging. Typical approaches like supervised fine-tuning or vanilla knowledge distillation (KD) often lead to suboptimal performance or even negative transfer. Our motivational experiments reveal key issues with vanilla KD methods: feature imitation suffers from redundancy and uneven recommendation ability across layers, while prediction mimicking faces conflicts caused by differing weight distributions of prediction heads. To address these challenges, we propose a simple yet effective framework, C2KD, to transfer task-relevant knowledge from two complementary dimensions. Specifically, our method incorporates: (1) cross-layer feature imitation, which uses a dynamic router to select the most relevant teacher layers and assimilate task-relevant knowledge from the teacher’s late layers, allowing the student to concentrate on the teacher’s specialized knowledge; and (2) cross-head logit distillation, which maps the intermediate features of the student to the teacher’s output head, thereby minimizing prediction discrepancies between the teacher and the student. Extensive experiments across diverse model families demonstrate that our approach enables 1B-parameter SLMs to achieve competitive performance compared to LLMs (e.g., Llama3-8B), offering a practical solution for real-world on-device sequential recommendations.

## introduction large language models (llms) @xcite have demonstrated great potential in sequential recommendation tasks @xcite , leveraging their extensive world knowledge and contextual reasoning. among them, generation-based llm recommenders @xcite @xcite are an important branch, which usually inherits the autoregressive architecture of llms, transform user behaviors into language prompts, and employ low-rank adaptation (lora) @xcite to repurpose pretrained llms for recommendation tasks. however, their large model size and high computational costs hinder deployment in resourceconstrained environments. for example, the representative recommender llara @xcite has over 7 billion (7b) parameters and takes 3.8 seconds to process each user sequence, highlighting the need for more efficient alternatives. small language models (slms) 1 have recently garnered attention for achieving comparable performance across various tasks with smaller model size @xcite @xcite , offering high efficiency and lower deployment costs. these strengths make slms wellsuited for real-world on-device applications, such as personalized e-commerce recommendations. despite their potential, the application of pretrained slms to sequential recommendation tasks remains largely underexplored, especially in their ability to achieve performance comparable to llms. to address this, we first conduct preliminary studies across various popular slms, as shown in figure 1 . the results reveal that directly applying supervised fine-tuning (sft) to slms on recommendation datasets results in significantly worse performance compared to llms, implying that @xcite . we follow the evaluation protocol in @xcite . the purple dashed line indicates teacher model performance. our approach significantly outperforms vanilla kd methods. sft alone is inadequate to equip slms with the capabilities needed for effective recommendations. to mitigate the performance gap while preserving the efficiency of slms, knowledge distillation (kd) offers a promising solution by transferring knowledge from large models to smaller models. while our study (figure 1 ) shows that vanilla kd methods @xcite , such as prediction mimicking ('vanilla logit kd') and feature imitation ('vanilla feature kd'), often fail to effectively close the performance gap between teacher and student models or even result in negative transfer. to investigate the limitations of vanilla kd methods for slm-based sequential recommendations, we conduct motivational experiments in section 2. we identify two key challenges that hinder effective task-relevant knowledge transfer: first, traditional prediction mimicking method suffers from distinct output head weight distributions. in this case, even when teacher and student features are similar, head disparities amplify their differences, which leads to contradictory learning and interferes with optimization. second, the final layer of the llm does not always deliver the best performance, and the contribution of each llm layer to recommendation abilities varies across different datasets. in light of these challenges, we hereby propose a simple yet effective kd framework to empower slm-based recommenders. given a well-trained llm as the teacher, our method transfers knowledge to pretrained slms by fine-tuning them with lora @xcite , updating only a small set of parameters. the task-relevant knowledge transfer is achieved through two complementary perspectives: (1) cross-layer feature imitation, which aligns the teacher's and student's intermediate features using a dynamic router, allowing the student to select the most relevant teacher guidance across different layers. moreover, to focus on taskrelevant knowledge, feature imitation is applied only to the teacher's late layers, with learnable filters extracting recommendation-related features for more efficient knowledge transfer. (2) crosshead prediction mimicking, which utilizes a learnable projection layer with orthogonal constraints to map the student's intermediate features into the teacher's prediction head. this process generates cross-head predictions that are then aligned with the teacher's original outputs, enabling more effective and harmonious knowledge distillation. notably, our framework is orthogonal to other llm post-training efficiency techniques, such as quantization and pruning, and can be combined for further improvements in efficiency. without bells and whistles, comprehensive experiments demonstrate that our method consistently exhibits superior performance across various recommendation datasets while maintaining high efficiency. for example, on three large-scale recommendation datasets @xcite @xcite , our method enables llama3-1b to achieve performance comparable to llama3-8b, despite having a model size less than 15% of llama3-8b and achieving a 4× inference speedup, as shown in figure 2 . furthermore, our method consistently outperforms existing kd methods, revealing the importance of designing tailored kd methods for building slmbased sequential recommenders foot_2 . ## motivational study in llm-based recommenders, a common approach is to apply lora @xcite on pretrained llms to adapt them to recommendation tasks. however, despite its effectiveness, the inference cost and computational burden of llms make them impractical for resource-constrained applications. on the other hand, slms, with their smaller size, are well-suited for on-device deployment but often suffer from limited performance when directly fine-tuned on recommendation tasks (as shown in figure 1 ). to strike a balance between effectiveness and efficiency, knowledge distillation (kd) provides an effective solution by transferring knowledge from large teacher models (llms) to smaller student models (slms). to assess the applicability of kd in building slm-based sequential recommenders, we revisit vanilla kd methods and reveal their limitations in this section. sequential recommendation. given a user's historical interactions in chronological order, a llmbased sequential recommender aims to select the item i p that is truly preferred by user u from the candidate set c @xmath0, where n is the number of candidates. following the llara framework @xcite , hybrid prompts that combine collaborative signals from id-based models (e.g., sasrec @xcite ) with item text descriptions are fed into the llm. item ids are mapped to tokens in the llm space using a trainable projector. the task-specific loss l task is defined in the form of causal language modeling: however, applying vanilla kd methods to student recommendation models often results in marginal performance gains or even negative transfer (as shown in fig. 1 ). we suspect these issues arise from two main factors: layer redundancy in the teacher model and the discrepancy between the output heads of the teacher and student models. for prediction kd, previous methods @xcite directly minimize the discrepancy between teacher and student predictions. however, since llms and slms are pretrained on different datasets, their output heads naturally inherit distinct distribution patterns and lie in different spaces. as a result, directly aligning predictions can be ineffective and may hinder the distillation of useful knowledge. for feature kd, previous layer-wise distillation methods @xcite use a linear mapping function to align teacher and student feature layers, assuming that performance improves gradually as layers deepen. however, this approach overlooks the potential layer redundancy, where certain intermediate llm layers contribute less to the recommendation task. to verify the above conjectures, we conduct preliminary studies on various recommendation datasets, as shown in fig. 3 . we employ a layerwise probing strategy to evaluate the recommendation ability of each decoder layer in llms and slms. in detail, we halt inference at each decoding layer and use learnable probing heads to map intermediate features to tokens, which are trained with l task . additionally, we apply pca projection to the output head weights of the teacher and student models to visualize the distribution differadditionally, the output head weight distribution gap between different models (cf) is examined to reflect the potential risk of using different heads for prediction mimicking. ences between them. based on these studies, we derive the following insights: ## method in this section, we introduce c2kd, a novel knowledge distillation based framework that combines cross-layer feature distillation and cross-head prediction mimicking for slm-based sequential recommendation. notably, we keep the pretrained slm backbone frozen and leverage lora modules to transfer task-specific knowledge from well-trained llms to slms. this approach significantly reduces training computation costs while avoiding catastrophic forgetting. ## conclusion this paper proposes a novel distillation framework that efficiently transfers task-relevant knowledge from llms to slms for sequential recommendation. our motivational study reveals the limitations of vanilla kd methods, including layer redundancy and uneven recommendation ability across llm layers, along with differing weight distributions in prediction heads. to address these challenges, our method leverages cross-layer feature imitation and cross-head logit distillation, enabling harmonious and effective task-relevant knowledge transfer. extensive experiments demonstrate that 1b-parameter slms can achieve performance comparable to 8bparameter llms, providing a practical and scalable solution for recommendation systems on resourceconstrained devices.