F ed C o T : Federated Chain-of-Thought Distillation for Large Language Models (2025)

Abstract: Large Language Models (LLMs) have emerged as a transformative force in artificial intelligence, demonstrating exceptional proficiency across various tasks. However, their deployment in resource-constrained environments and concerns over user data privacy pose significant challenges. In contrast, Small Language Models (SLMs) offer computational efficiency but often lag in performance. To address these issues, we propose FedCoT, a federated framework designed for the Chain-of-Thought (CoT) distillation of knowledge from LLMs to SLMs, while ensuring the preservation of clients’ data privacy. FedCoT ensures secure and efficient knowledge transfer from an LLM on a high-powered server to an SLM on a resource-constrained client, while adhering to privacy requirements. Leveraging perturbed prompts and rationales generated through the CoT approach, the framework enhances the performance of the client’s SLM without compromising user data privacy within a multi-task learning framework. We propose two privacy protection strategies: the Exponential Mechanism Strategy and the Adaptive Exponential Mechanism Strategy, which balance user prompt privacy and the usability of rationales. Empirical evaluation on various text generation tasks demonstrates the effectiveness of FedCoT in training task-specific SLMs with enhanced performance while prioritizing data privacy protection. Our code has been contributed to the FATE open-source project and is now publicly accessible at

## introduction large language models (llms) have risen as a revolutionary force in artificial intelligence. prominent llms, such as gpt-4 (openai, 2023), llama @xcite , and qwen @xcite , have garnered the attention of researchers and practitioners alike, demonstrating unparalleled proficiency across numerous tasks. nevertheless, the sheer size of these models presents significant obstacles for real-world deployment, particularly in environments with limited resources @xcite @xcite . meanwhile, as llms gain escalating popularity and widespread utilization, privacy concerns have moved to the forefront, especially when it comes to user data and llms inference. in contrast, small language models (slms) often exhibit superior computational efficiency and faster convergence rates, rendering them perfectly suited for real-time applications or resource-constrained environments. nonetheless, slms also possess certain drawbacks stemming from their performance limitations. the question then arises: how can we effectively combine the predictive prowess of llms with the nimbleness of slms, all while adhering to privacy requirements? to address these challenges, we propose fedcot, a federated framework designed for the chain-of-thought (cot) @xcite distillation of knowledge from llms to slms, while ensuring the preservation of clients' data privacy. fedcot ensures secure and efficient knowledge transfer from an llm on a high-powered server to an slm on a resource-constrained client. the challenge lies in maintaining the privacy of client data while leveraging the server's llm to aid in training the client's slm for text generation tasks, thereby elevating its performance. fedcot aims to bridge this gap, enabling secure and efficient knowledge transfer between llm and slm, and ultimately enhancing the capabilities of the slm without compromising privacy. as illustrated in figure 1 (a), within our framework, the process works as follows. initially, the client transmits perturbed prompts to the server's llm. these prompts are protected by the fedcot prompt encoder, which employs differential privacy (dp) principles @xcite , ensuring privacy protection. subsequently, the server's llm generates perturbed rationales from these prompts through the cot approach and relays them back to the client. upon receiving these perturbed rationales, the client's rationales decoder reconstructs them into their original, aligned form corresponding to the raw prompt. ultimately, the client utilizes cot knowledge distillation @xcite to train its task-specific slm. this process leverages both label data and rationales within a multi-task learning paradigm @xcite @xcite . these rationales justify the predicted labels and serve as insightful guidance for training smaller and domain-specific models. previous endeavors to incorporate dp into language models, specifically through dp-sgd @xcite , have primarily centered on navigating the delicate balance between utility and privacy. this is achieved by introducing calibrated noise into gradients or text representations during the model training process. nonetheless, these methods inherently rely on a trusted server to gather data from data owners for model training @xcite , significantly limiting their applicability in scenarios where such trusted servers are not available, as is the case in our research context. within the fedcot framework, to achieve a balance between preserving the privacy of user prompts and enhancing the usability of rationales, we introduce two privacy protection strategies: the exponential mechanism strategy and the adaptive exponential mechanism strategy. in the exponential mechanism strategy, we utilize an exponential mechanism to obfuscate the prompts @xcite @xcite , followed by decoding the perturbed rationales through in-context learning (icl) @xcite . in the adaptive exponential mechanism strategy, we utilize an encoder-decoder slm specifically designed to encode original prompts into perturbed prompts and subsequently decode perturbed rationales back into their original form. to effectively train this unified encoder-decoder slm, we utilize a multi-task learning paradigm @xcite , encompassing both the encoding and decoding training processes. our contributions are summarized as follows: the strategies effectively balance user prompt privacy and the usability of rationales. ## the proposed fedcot framework in this section, we introduce fedcot, a federated framework designed for the cot distillation of knowledge from llms hosted on a high-powered server to slms deployed on a resource-constrained client. the fedcot framework can enhance the performance of slms while maintaining client data's privacy, leveraging the capabilities of llm. we assume the server to be semi-honest, implying that it may attempt to recover the private data of the client from the information it observes. we illustrate the fedcot in figure 1 (a), outline its training algorithm in algorithm 1, and detail its resource requirements in appendix a. algorithm 1 fedcot input: t : total number of rounds; p: encoding training datasets; r: decoding training datasets; d: task-specific training datasets; η ϕ : learning rate of encoder-decoder slm; η ω : learning rate of task-specific slm. output: g ϕ , f ω . 1: ▷ multi-task training for encoder-decoder slm based on public datasets p and r. 2: for each epoch @xmath0: ## conclusions in this study, we introduce fedcot, a federated framework designed to distill knowledge from llms to slms in resource-constrained environments. fedcot facilitates secure knowledge transfer from llms to slms by leveraging perturbed prompts and rationales, thereby enhancing the performance of slms without compromising user privacy. we present two innovative privacy protection strategies, including an adaptive exponential mechanism strategy, which effectively balance privacy preservation and the usability of rationales. experiments on various text generation tasks demonstrate fedcot's ability to enhance slm performance with llm support while prioritizing data privacy.