T aylor Unswift: Secured Weight Release for Large Language Models via T aylor Expansion (2024)

Abstract: Ensuring the security of released large language models (LLMs) poses a significant dilemma, as existing mechanisms either compromise ownership rights or raise data privacy concerns. To address this dilemma, we introduce TaylorMLP to protect the ownership of released LLMs and prevent their abuse. Specifically, TaylorMLP preserves the ownership of LLMs by transforming the weights of LLMs into parameters of Taylor-series. Instead of releasing the original weights, developers can release the Taylor-series parameters with users, thereby ensuring the security of LLMs. Moreover, TaylorMLP can prevent abuse of LLMs by adjusting the generation speed. It can induce low-speed token generation for the protected LLMs by increasing the terms in the Taylor-series. This intentional delay helps LLM developers prevent potential large-scale unauthorized uses of their models. Empirical experiments across five datasets and three LLM architectures demonstrate that TaylorMLP induces over increase in latency, producing the tokens precisely matched with original LLMs. Subsequent defensive experiments further confirm that TaylorMLP effectively prevents users from reconstructing the weight values based on downstream datasets.

## introduction training large language models (llms) is an expensive and complex endeavor, requiring substantial investments in financial and computational resources @xcite . in particular, it requires a huge collection of high-quality datasets from diverse domains, which proves to be labor-intensive and time-consuming. however, once released, the llms may face the risks of abuse such as unethical or commercial exploitation @xcite ., * equal contribution. 2021). this raises the critical and urgent challenge of ensuring the security of released llms. in this work, we explore the security aspects of existing mechanisms for releasing llms. this process involves developers providing access to their models for general users, with criteria restricting access to ethical and non-commercial purposes @xcite . currently, there are two primary mechanisms for releasing llms: api release @xcite and open-source @xcite . with the release of apis, user authorization is managed through specific api keys. examples of this mechanism include the chat-gpt @xcite , gemini @xcite , and claude models @xcite . in these cases, users do not have access to the architectures or weights of the llms. instead, they share their private data to the developers and receive the processed results, as shown in figure 1 (a) . consequently, the api release mechanism may cause the data privacy concerns @xcite . on the other hand, the open-source mechanism fully shares the llm weights with users, as shown in figure 1 (b) @xcite . common examples include llama @xcite , @xcite , and phi @xcite . while the open-source mechanism ensures the safety of users' private data, it also raises significant challenges for developers. specifically, the open-source mechanism can break the ownership of llms, as users gain control over the models and can use them for any purpose, even those prohibited by the developers. for example, users may exploit llms for unethical or commercial purposes, both of which may be prohibited by the developers. this potential loss of control and ownership may lead many model developers to avoid sharing their models @xcite . therefore, there is a dilemma between protecting ownership rights and ensuring open access to llms, posing a significant challenge for developers. can we solve the access dilemma for llms? we propose taylor-series mlp (taylormlp) to protect the ownership of released llms and prevent their potential abuse. as illustrated in figure 1 (c), taylormlp addresses the dilemma by securing the weights of llms into latent parameters. by sharing these parameters instead of the original weights with users, developers can maintain ownership of their models while allowing users to harness the model's performance. specifically, taylormlp converts the original weights into parameters of the taylor series. our empirical experiments confirm that it is infeasible to reconstruct the original weights from these taylor series parameters, thereby ensuring the security of the model's parameters and allowing safe access to its functional capabilities without full model exposure. can we prevent unauthorized users from exploiting the llm for their own purposes? to prevent unauthorized users from abusing the protected llms, taylormlp allows developers to control the utility of the llm by adjusting the speed of token generation. specifically, taylormlp induces low-speed token generation for the secured llms by increasing the terms in the taylor-series. it significantly increases the number of floatingpoint operations required for the generation process, leading to a notable increase in latency. our empirical studies show that taylormlp induces more than 4× increases in latency, while maintaining the produced tokens precisely matched with original llms. this intentional delay helps developers prevent the potential large-scale unauthorized use of their released llms. ## architecture of transformers we focus on transformer-based llms @xcite to develop the method of llm protection. a transformer block consists of an attention layer and mlp layer. an mlp layer is a pipeline of a linear layer, activation function, and another linear layer, whose architecture is shown in figure 2 (a). let v, b and w, c denote the weight matrixes of the two linear layers. given the input tensor x to the mlp layer, the output value in the i-th dimension is given by where @xmath3; ⟨•, •⟩ denotes the inner product; and w i and c i denote the i-th row and i-th element of w and c, respectively. in this work, we introduce a method to secure the weights of mlp layers within the transformer architecture. this can be widely applied to powerful llms, such as llama @xcite , phi @xcite , and gemma @xcite . the framework of taylormlp is shown in figure 2 (b). intuitively, taylormlp transforms the weight matrices b, w and c into latent parameters θ i,0 ,• • •,θ i,n . in this way, taylormlp generates output tokens without needing the original weight values. in this section, we first describe the weight transformation of taylormlp. then, we demonstrate the outputs of taylormlp theoretically converges to the original mlps. finally, we discuss the benefits of taylormlp to llm community. ## taylormlp benefits llm community taylormlp benefits llm community by protecting the ownership of developers, preventing abuse of llms, defense against user fine-tuning, and ensuring secure uses of llms. ## experiments in this section, we conduct experiments to evaluate taylormlp by answering the following research questions: rq1: can taylormlp retain the accuracy of original llms while adjusting generation speed? rq2: how does taylormlp defend against fine-tuning on downstream datasets and distilling on large-scale datasets? rq3: how does the expansion order influence the output of taylormlp compared with that of original llms? ## related work file encryption. encryption technologies can effectively prevent the unauthorized use of digital files, including the checkpoint file of large language models (llms). representative algorithms include the advanced encryption standard (aes) @xcite , and rivest-shamir-adleman (rsa) @xcite . however, these methods are not suitable for llms because the models cannot perform inference using encrypted weight values. this limitation prevents users from harnessing the capabilities of released llms on their private datasets, which may cause a loss of market opportunities. offsite-tuning. the offsite-tuning method allows developers to share parts of large language models (llms) with users @xcite . however, this method has several disadvantages. first, users need to perform fine-tuning, which is extremely costly and time-consuming. additionally, fine-tuning llms is challenging for individuals who are not machine learning engineers. these factors hinder the application of offsitetuning methods in real-world scenarios. context: new york (cnn) -more than 80 michael jackson collectibles -including the late pop star's famous rhinestonestudded glove from a 1983 performance -were auctioned off saturday, reaping a total @xmath0120,000 in sales. taylormlp @xmath4: _refascus bout bout bout bout bout bout bout bout • • • bout taylormlp @xmath5: the article was written from the article from • • • from the article taylormlp @xmath6: here are the answers to your questions: 1. where was the auction held? the auction was held at the hard rock cafe in new york's times square. 2. how much did they make? the auction raised a total of @xmath1120,000. 4. who bought the jackson glove? the jackson glove was bought by hoffman ma, who purchased it on behalf of ponte 16 resort in macau. 5. where was the buyer of the glove from? the buyer of the glove, hoffman ma, is from hong kong, china. taylormlp @xmath7: according to the article, the auction was held at the hard rock cafe in new york's times square. the auction made a total of @xmath2120,000. the michael jackson's famous rhinestone-studded glove from a 1983 performance was sold for $420,000 to a buyer from hong kong, china, who is hoffman ma, representing ponte 16 resort in macau. kl div 0 0.15 0.3 0.45 0.6 0.75 0.9 rouge-1 rouge-1 figure 5: the kl divergence of output probability and rouge-1 score versus the expansion order. ## conclusion in this work, we propose taylormlp to preserve the ownership of released llms and prevent their abuse. specifically, taylormlp preserves the ownership of llm by transforming the weights of llms into parameters of taylor-series. instead of releasing the original weights, developers can release the taylor-series parameters with users, thereby ensuring the security of llms. defensive experiments confirm that taylormlp effectively prevents users from reconstructing the weight values based on downstream datasets. moreover, tay-lormlp prevents abuse of llms by inducing lowspeed token generation for the protected llms. this intentional delay prevents the potential largescale unauthorized abuse. empirical and case studies show that taylormlp significantly increases the latency of token generation, while maintaining the chat capabilities and performance on downstream tasks. both qualitative and quantitative results demonstrate the effectiveness of taylormlp in ensuring the security of the released llms. this indicates its potential in real-world applications.