CR a S h: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model (2023)

Abstract: Instruction tuning has recently been recognized as an effective way of aligning Large Language Models (LLMs) to enhance their generalization ability across various tasks. However, when tuning publicly accessible, centralized LLMs with private instruction data, privacy concerns are inevitable. While direct transfer of parameterized modules between models is a plausible approach to address this, its implications and effectiveness need further exploration. This paper focuses on Offsite-Tuning (OFT), a representative technique that transfers transformer blocks between centralized LLMs and downstream emulators. Given the limited understanding of the underlying mechanism of OFT, we perform an empirical analysis on LLMs from the perspectives of representation and functional similarity. Interestingly, our findings reveal a unique modular structure within the layers of LLMs that appears to emerge as the model size expands. Simultaneously, we note subtle but potentially significant changes in representation and intermediate predictions across the layers. Inspired by these observations, we propose CRaSh, involving Clustering, Removing, and Sharing, a training-free strategy to derive improved emulators from LLMs. CRaSh significantly boosts performance of OFT with billions of parameters. Furthermore, we investigate the optimal solutions yielded by fine-tuning with and without full model through the lens of loss landscape. Our findings demonstrate a linear connectivity among these optima falling over the same basin, thereby highlighting the effectiveness of CRaSh and OFT.

## introduction nowadays, there is a growing interest in large language models (llms) such as palm @xcite , llama @xcite and gpt-4 (openai, 2023) due to their potential towards advanced intelligent systems. by employing techniques like prompt learning @xcite and instruction tuning @xcite @xcite , the behavior of llms can be aligned with human intent using a small amount of data. @xcite @xcite . however, the centralization of llms poses a significant challenge concerning the trade-off between high performance and user data @xcite . for instance, openai provides fine-tuning apis * that allows users to upload personal data for further fine-tuning of davinci models family, which has gained popularity in the industry, like ope-nai @xcite , google bard † , and anthropic claude ‡ . safeguarding the privacy of both llms and downstream user data is an urgent concern. one viable approach involves the direct transfer of parameterized modules between models, such as federated learning (fl) @xcite and split learning (sl) @xcite , where limited exploration are conducted on llms with billions of parameters. recently, @xcite propose offsite-tuning (oft) for transfer learning that operates independently of full llms with sizes exceeding 1b. oft entails compressing the llm into a smaller model known as emulator by layer dropping, followed by fine-tuning the emulator using user data. finally, the parameterized modules are transferred from emulator and seamlessly integrated into llm in a single turn. despite the promising results obtained by oft, there is still limited understanding of its underlying mechanism. in this paper, we conduct a detailed analysis of llms from the perspective of representation and functional similarity @xcite to enhance our understanding of oft. through comparing the similarity of hidden states across layers, we observe emergence of modular structure of representations within llms. as depicted in figure 1 , models with a size less than 10b exhibit uniform representations across all layers. however, for models of size 13b and 30b, modular structure becomes apparent in the representation similarities. for the 13b model, high similarities are observed between layers 5 and 20, forming a light-colored block-diagonal structure (i.e., modular structure), as well as between layers 25 and 40. additionally, we note subtle changes in the representation between adjacent layers. we further analyze functional similarity to explore the intermediate predictions of each layer, which enhances these findings. building upon our findings, we propose a completely training-free strategy to enhance fine-tuning without relying on full model. this strategy consists of three steps, namely clustering, removing, and sharing (crash). in the initial step, we cluster adjacent layers based on their similarity using various hierarchical clustering algorithms @xcite . we then remove layers within the same cluster to obtain an emulator. the remaining layers are shared as a supplement to the removed layers. finally, selected layers of the emulator are updated using downstream data and transferred to be seamlessly integrated into llms, resulting in improved performance. through the utilization of crash, we significantly enhance the performance of fine-tuning llms without full models across multiple datasets. in order to comprehend the relationship of optimal solutions in crash, we visualize the optima using loss surfaces and mode connectivity @xcite . this study offers valuable insights into the effectiveness of crash. in summary, our main contributions can be summarized as follows: ## empirical analysis in this section, we analyze the inherent similarity of layers in llms from two complementary perspectives: (i) representation similarity (section. 2.1), which examines the differences in activations among intermediate layers, and (ii) functional similarity (section. 2.2), specifically the variations in predictions among intermediate layers. ## methodology empirical analysis of llms can inspire fine-tuning approaches, contributing to privacy protection for both centralized llms and user data. in this section, we begin by revisiting offsite-tuning (oft), a representative method for fine-tuning llms without full model. subsequently, we introduce our proposed crash method inspired by section. 2. ## discussion loss landscape and mode connectivity. in order to comprehend the effectiveness of crash, we conduct an analysis of the relationship between optimal minima derived from plug-in and full model fine-tuning. firstly, based on figure 6a , we observe that the initialization resides in a low basin and performs well across various datasets, which benefits from the over-parameterization and generalization of llms. consequently, by optimizing the llm at a relatively low cost on the target dataset, it can effectively perform on this dataset. this observation highlights the efficacy of delta-tuning @xcite and offsite-tuning. secondly, the solutions derived from crash and full fine-tuning reside within the same basin. utilizing the clustering and sharing steps, crash exhibits a closer proximity to full fine-tuning compared to oft (refer to figure 6b ). finally, through the interpolation of various solutions, we observe their mode connectivity in the parameter space, wherein crash exhibits a smoother transition towards full finetuning compared to oft, which undergoes more dramatic changes. based on the visualization of the loss landscape, there is potential to enhance crash for better performance in future directions. ## related work delta-tuning @xcite ) methods efficiently update a subset of parameters compared to the entire llms @xcite @xcite . however, these methods require feeding data into the entire llms, which is not resourceefficient and raises concerns about the privacy of llms. black-box tuning @xcite , as applied to centralized llms, attempts to learn parameters based on input or output text @xcite , which helps protect llms but poses risks to user data. directly manipulating model parameters instead of transferring data has found wide applications in federated learning @xcite , split learning @xcite , distributed gradient descent @xcite @xcite , branch-train-interpolation @xcite , and collaborative machine learning development @xcite . these methods either involve model sharing between servers and clients or require multi-turn communication to achieve coverage, which poses limitations when applied to llms with billions of parameters. offsite-tuning @xcite is a notable method that addresses these challenges by selectively sharing layers of llms with billions of parameters between servers and clients in a single turn. on the other hand, previous studies have focused on gaining insights into the intermediate representation of neural networks for better fine-tuning @xcite @xcite . in terms of transformers, @xcite observed clustering of layer representations in fine-tuned bert models @xcite , supporting the notion of layer redundancy @xcite . in contrast, we find that clustering also emerges in pre-trained llms as the model size increases, which helps offsite-tuning on llms. a detailed introduction to related work is presented in appendix d. ## conclusion in this paper, we uncover block structure of representation within the intermediate layers of llms, indicating the clustering of layers in depth models. based on these observations, we propose a completely training-free strategy to enhance fine-tuning without full model. the strategy consists of three steps: clustering, removing, and sharing (crash). crash boosts the performance of offsite-tuning for llms on various datasets. further analysis of the loss surface and mode connectivity provides insights into the effectiveness of crash.