Extracting Prompts by Inverting LLM Outputs (2024)

Abstract: We consider the problem of language model inversion: given outputs of a language model, we seek to extract the prompt that generated these outputs. We develop a new black-box method, output2prompt, that extracts prompts without access to the model’s logits and without adversarial or jailbreaking queries. Unlike previous methods, output2prompt only needs outputs of normal user queries. To improve memory efficiency, output2prompt employs a new sparse encoding techique. We measure the efficacy of output2prompt on a variety of user and system prompts and demonstrate zero-shot transferability across different LLMs.

## introduction given outputs of a large language model (llm), is it possible to extract the prompt that generated these outputs? if the llm is wrapped into an api or app that automatically prepends a "system prompt" to all user queries, is it possible to extract this system prompt by interacting with the api? this problem is known as language model inversion @xcite . the current state-ofthe-art inversion method is logit2prompt, which extracts inputs to the model given its logits, i.e., next-token probability distribution @xcite . logit2prompt cannot be applied to many llms, however, because their apis do not reveal their logits @xcite . even when a model's logits are available (or can be inferred), inversion using logit2prompt can be prohibitively expensive, with hundreds of thousands of queries required to extract a single prompt. another approach is to steer the model into outputting its context, including the system prompt, via specially crafted adversarial queries @xcite . this technique is not stealthy because adversarial queries are different from normal user queries. it is also brittle and model-specific because its efficacy depends on the target model's instruction-following capabilities, lack of safety alignment, and the absence of defenses such as protection prompts added to system prompts and input and/or output filters. finally, adversarial extraction is simply not possible in deployments that limit users to pre-defined queries (e.g., when the target llm acts as an assistant for a fixed task). our contributions. we design, implement, and evaluate output2prompt, a new prompt extraction method that uses only the text outputs of llms generated in response to normal user queries. output2prompt does not require access to logits, nor adversarial queries. we evaluate output2prompt on a variety of user and system prompts, including those of realworld gpt store apps (gpts). it outperforms prior methods, including logit2prompt (morris et al., 2023b)-without access to logits and with two orders of magnitude fewer training samples-achieving cosine similarity of 96.7 compared to 93.5 by logit2prompt. unlike prior extraction methods, output2prompt transfers across different language models (both base and instructiontuned) with little loss in performance, maintaining cosine similarity above 92. even when prompts extracted by output2prompt are different from the original prompts, we show that the extracted prompts are semantically close via high cosine similarity and empirical metrics such as asking an llm whether the prompts are functionally similar. prompt extraction can thus be used to clone llmbased apps (e.g., gpts) without any adversarial queries. ## threat model we consider the scenario where a hosted large language model (llm) is available to users via a standard web api. the api returns a single output in response to each user prompt. if the user calls the api multiple times with the same prompt, they may obtain different outputs. the user may also access the api indirectly, via an application such as a gpt store app @xcite . the api, or an app running on top of the api, may prepend a system prompt to each user prompt. in our evaluation, we provide the system prompt as the first ("assistant") turn and the user's prompt as the second ("user") turn. different turns have different privilege levels @xcite denoted by special tokens, which are managed by the system and cannot be edited by the user. we assume that the system will not reveal the system prompt, nor any information about the internal state of the llm (such as logits @xcite ), nor any additional apis (such as logit bias) that could help infer this state @xcite . this assumption matches typical hosted llm deployments. our threat model covers several realistic scenarios. in the first scenario, the adversary observes outputs published by other users of the api and wants to extract the prompts they used. in the second scenario, the adversary accesses the system directly and wants to extract the hidden system prompt. in this paper, we develop an extraction method that is stealthy and non-adversarial. the adversary's queries (if any) should be indistinguishable from the queries of non-adversarial users. stealthiness further requires that the attack be local. the adversary cannot rely on an auxiliary llm as an oracle to either generate adversarial queries, or help extract prompts from observed outputs. the attack should also be robust with respect to any defenses deployed by llms and llm-based apps. target llms may block adversarial and "jailbreaking" queries via query filtering, and/or employ internal safeguards, such as safety alignment, to try and avoid disclosing their system prompts. success of the attack should not depend on the efficacy of these defenses. attacks that use llms as oracles fundamentally depend on these llms' (a) capability, and (b) availability to generate adversarial queries and/or perform adversarial inference in response to the adversary's instructions. both (a) and (b) are opaque, not controlled by the adversary, and can-and do-change over time (e.g., a new version of the llm may refuse to assist in prompt extraction). therefore, oracle-dependent attacks are not robust. we show an example of how adveresarial queries may fail in appendix f. we operate in an even more stringent threat model. the adversary is limited to non-adversarial queries only, specifically queries that are on the model's pre-defined list of supported queries (e.g., for customer service assistance), or else suggested by the model itself when the user asks it for suitable queries. by definition, such queries are indistinguishable from the queries issued by typical users. ## methodology we define prompt extraction as a language model inversion @xcite problem: given black-box access to an llm and multiple outputs of this llm based on an unknown prompt, generate the prompt. the key difference with @xcite ) is that we do not assume access to any of the llm's internal state (in particular, logits). this variant of the problem is called "sample inversion" in @xcite . llm outputs become inputs into the inversion model in this setup. ## related work model inversion. there is a large body of work on model inversion, i.e., inverting the outputs of learned models @xcite @xcite @xcite . model inversion is distinct from model extraction or stealing: the former aims to reconstruct model inputs, the latter model parameters. recent work in nlp has shown that text inversion from model outputs is possible in some scenarios: (morris et al., 2023a) extract inputs from text embedding vectors, while (morris et al., 2023b) extract inputs from llm outputs. the latter method requires access to logits, which is not available in many real-world llm deployments @xcite . @xcite characterize prompt design as an output inversion problem and use white-box access to the model to perform gradient descent on soft prompts, then convert soft prompts to hard prompts. @xcite apply a classifier to model outputs to determine the type of the prompt, then use chatgpt as an oracle to reconstruct the prompt. this method relies on chatgpt's "remarkable reversing ability" and is neither local (see section 2), nor controlled by the adversary (e.g., it cannot work if chatgpt refuses model-inversion queries). ## conclusion we designed, implemented, and evaluated output2prompt, a stealthy, black-box, transferable method for extracting prompts by inverting llm outputs. unlike prior work, output2prompt does not require the target llms's logits, nor does it rely on adversarial or jailbreaking queries (and thus cannot be thwarted by defenses). our results demonstrate that many user and system prompts are intrinsically vulnerable to extraction. our new sparse encoder technique may have applications in other settings where llms operate on many independent inputs during training and inference.