L attice G en: Hiding Generated Text in a Lattice for Privacy-Aware Large Language Model Generation on Cloud (2024)

Abstract: In the current user-server interaction paradigm of prompted generation with large language models (LLMs) on cloud, the server fully controls the generation process, which leaves zero options for users who want to keep the generated text private to themselves. For privacy-aware text generation on cloud, we propose LatticeGen, a cooperative protocol in which the server still handles most of the computation while the client controls the sampling operation. The key idea is that the true generated sequence is mixed with noise tokens by the client and hidden in a noised lattice. Only the client knows which tokens are the true ones. Considering potential attacks from a hypothetically malicious server and how the client can defend against it, we propose the repeated beam-search attack and the mixing noise scheme. In our experiments we apply LatticeGen to protect both prompt and generation. It is shown that while the noised lattice degrades generation quality, LatticeGen successfully protects the true generation to a remarkable degree under strong attacks (more than 50% of the semantic remains hidden as measured by BERTScore).

## introduction many of the high-performing large language models (llms) need to be deployed on cloud servers, whether they are open-sourced but have an intensive need for computation @xcite @xcite , or behind a paywall like @xcite . this raises new privacy challenges @xcite @xcite , since users have to send or receive their data to/from cloud providers. in this work we focus on a popular interaction paradigm between end users and a server hosting an llm on cloud named prompted generation: the user sends server a prompt, which is usually an instruction @xcite or the beginning * equal contribution. both are corresponding authors. as llms become widely deployed in professional and social applications, we argue that in prompted generation, there are many scenarios in which not only the prompts, but also the generated texts need some level of obfuscation, because they can directly affect the user's real-life private decisions. for example, a customer is likely to go to the restaurant suggested by the llm, and a writer could take inspiration from outputs provided by the llm. with the goal of preventing the server from gaining complete knowledge of the generated text and prompt, we propose latticegen (figure 2 ), a client-server interaction protocol in which the user and client conduct privacy-aware generation token-by-token in a cooperative way. the protocol can be executed by a local client so that the interface is kept simple for the user. we summarize our key contributions below: the server does llm inference and sends client back the next-token distributions for all n tokens, which are used by the client to sample the true and noise tokens for the next time-step. ## latticegen to prevent the server from gaining full knowledge of the generation and prompt, latticegen makes several core changes to the client-server interaction. on a high level, the server who possesses the llg-finetuned llm p l (the finetuning is detailed in §a) still handles most of the computation, while the client controls the token sampling operations and expands the lattice to the next time-step. in particular, the client will sample one true token and n -1 noise tokens, where @xmath0. in the end, both the server and client obtain the same noised lattice w n t , but only the client knows which token is the true one for each time step. in the beginning, the server needs to share the vocabulary v with the client, but all other parameters or configurations of the llm are not shared. we describe the protocol below. ## attack and defense in this section, we discuss potential attack algorithms from a hypothetically malicious server to decode the true token sequence {w true t } t @xmath1, and the client's noise generation schemes as defense. for notational simplicity, we will assume unigram (@xmath2, and the extension to @xmath3. we first establish metrics to measure the strength of attacks. metrics given a lattice w n t , the attacker's target is to decode a hypothesis sequence ŵ with ŵt ∈ {w 1 t , ..., w n t } having biggest overlap with the true generation w true . we define a simple true-ratio metric to measure the strength of the attack: in the repeated beam search attack to be described below, the result of the attack algorithm is not only one but n sequences { ŵi } n @xmath4.e., { ŵi t } n @xmath5. in this case, we argue that the defending noise scheme should prevent any of the hypothesis from having a high overlap with the true sequence, and measure it with the max-true-ratio: it should be clear that 1 n is a lower bound for max-true-ratio for any noise scheme, which provides an intuition of why larger n would better protect the true sequence. albeit intuitive, a big weakness of the true-ratio metric is that it only considers exact matches and 4 the average of the true-ratio will always be 1 n because each true token is in one of the n hypotheses. does not reflect the semantic similarity between the hypothesis and the true generation. therefore, in our experiments we will also use an embeddingbased metric bertscore @xcite to measure the leaked information on semantics. similar to true-ratio, bertscore is larger than zero and has a maximum value of 1 (we refer readers to its paper for details). we define max-bertscore in the same fashion as max-true-ratio and we omit the formulation for brevity. ## related work existing work in privacy-aware natural language processing (nlp) @xcite mostly focuses on protecting user data for training @xcite or inference, and the majority of works focus on natural language understanding (nlu) tasks @xcite . to the best of our knowledge, our work is the first to consider decoding-time privacy for llm prompted generation on cloud. lattice in nlp lattice @xcite is a graphical structure widely used in structured prediction problems to represent a range of hypotheses. in this work we adopt a simple linear-graph form of lattice which is known as the confusion network @xcite . the lattice structure has found interesting applications in neural nlp models. as a pioneering work, @xcite proposes lattice-based rnn encoders for machine translation, where the lattice is generated by merging results from different segmenters. @xcite proposes a neural lattice language model, which constructs a lattice of possible paths (segmentations) through a sentence in order to model multiple granularities. lattice-bert @xcite trains llm to predict a masked portion of a lattice representing possible segmentations of a sentence. to the best of our knowledge, our work is the first to utilize the lattice structure for privacy-aware generation. prompt anonymization contemporary and independent of our work, @xcite proposes to anonymize the named entities (e.g., change usa @xmath6, and de-anonymize after receiving the generated text from server. in comparison, latticegen offers a more general option in that all types of tokens, especially the generated tokens, can be noised. due to lack of space, we discuss related work on differential privacy, homomorphic encryption in §c. ## conclusion latticegen aims for an ambitious and seemingly conflicting goal: the server still does most computation for the generation but does not know what exactly is generated. this is achieved by our proposed noised lattice structure, and a cooperative generation protocol between the server and client. while the noised lattice degrades generation quality and inference speed, latticegen with our proposed mixing noise scheme successfully prevents a malicious server from recovering the true generation to a remarkable degree (more than 50% of the semantic remains unknown as measured by bertscore). we hope our work could inspire more research into this under-studied yet important field of privacy-aware llm generation on cloud.