Paper: Language Model Inversion (ICLR-2024)

Reviewer comment (Novelty):
Given the high similarity between [1] and this submission, every nook and cranny of their differences must be disclosed and analyzed to make the submission's contributions clear. I found comparisons lacking with many unanswered questions like: why not use the iterative version, which is shown to be extremely useful in [1]?

Author response:
> Given the high similarity between [1] and this submission, every nook and cranny of their differences must be disclosed and analyzed to make the submission's contributions clear. I found comparisons lacking with many unanswered questions like: why not use the iterative version, which is shown to be extremely useful in [1]?

We agree that the method here is not novel. We apply ideas from past inversion work including [1] to a different application domain (inverting LM outputs). The contributions are developing a new architecture, a dataset of LM prompts, new baselines, and an algorithm for making this practical to run in an API setting with logit-bias.

We tested the iterative refinement approach in [1] but unfortunately did not find it worked when the input is LM logits. We have added these negative results to the appendix, along with a small experiment explaining why our corrector module might not work recursively (due to a narrow range of BLEU scores in the training distribution).

Reviewer comment (Writing):
The bisection-based estimator seems effective, but I find it a bit abrupt and lacking an explanation.

Author response:
> The bisection-based estimator seems effective, but I find it a bit abrupt and lacking an explanation.

We’ve completely rewritten Section 5 in an effort to make our logit recovery algorithm more clear, and smooth the transition between sections. We plan to release our algorithm as a Python package to allow others to recover full probability vectors from argmax-only API access.

Reviewer comment (Experiments):
The method does not seem to consider the setting in which the model probabilities are not available at all (i.e., generation-only APIs), which is maybe the most relevant setting for extracting prompts given that the most powerful LLMs are private.

Author response:
> The method does not seem to consider the setting in which the model probabilities are not available at all (i.e., generation-only APIs), which is maybe the most relevant setting for extracting prompts given that the most powerful LLMs are private.

Sorry for the confusion. We *do* consider this baseline – “Sample Inverter” is a T5 encoder-decoder trained to directly predict a prompt from LLAMA-2 Chat outputs. We have edited the paper to make this more clear in both the theoretical discussion (Section 3.2) and presentation of results (Section 7).

Additionally, the algorithm in Section 5 (originally known as “Distribution Extraction”) is proposed for precisely this reason: it enables logits access to models when APIs only provide argmax directly, by use of the logit bias argument. (As of November 2023, logit bias and argmax output are available in OpenAI, Anthropic, Cohere APIs.)

Reviewer comment (Theory):
The method seems rather ad-hoc and not theoretically motivated. In the case of access to the conditional probability distribution, the method boils down to feeding (a transformed version of) the distribution back to a language model.

Author response:
We have added a new Subsection 3.1 (“Logits Contain Residual Information”) providing empirical and theoretical motivation why language model logits may reveal information about past tokens.

In terms of the method, we feel we are following standard practice in modern deep learning using Transformers. The main architectural challenge is how to feed in a high-dimensional vector (32,000 numbers) into a transformer. Main contributions were collecting a large dataset of prompts, extracting next-token probability distributions, and demonstrating that inversion is possible and effective in a variety of settings. We are not sure what would make a deep learning approach more theoretically motivated.

Reviewer comment (Writing):
In the beginning of section 4 where it’s stated that you “train on samples from the conditional model”, what is “the conditional model”?

Author response:
This line was simply meant to indicate that we train the inversion model on output from a (conditional) language model. It was not a critical point, and we’ve removed the statement entirely.

Reviewer comment (Theory):
Given that the vector v was projected from R^d to R^v (as part of the final linear layer of the generation model), the argument that projecting it back to R^d would lead to a loss of information feels a bit strange.

Author response:
This is a small notational inconsistency. The original dimensionality d is that of the victim language model (4096 for LLAMA) while the final dimensionality is the input embedding of the inverter language model (typically 768 for T5). So in this case, projecting directly from LLAMA to T5 space would lead to a loss of dimensionality with a factor of 4096/768≈5.3.

Reviewer comment (Theory):
Where is the theoretical discussion/experiments corresponding to inversion when only the text output is available? The methodology discussed in section 5 still assumes access to probabilities from the model, which are often not available in “models as a service” platforms.

Author response:
We did consider this baseline, sorry for not making it more clear. For Llama2-chat, we train a model to predict the prompt input given language model output samples . This is shown in Table 1 as “Sample Inverter”. We add an additional paragraph of theoretical discussion in this matter to Section 3.2 (“Inverting from outputs”).

Reviewer comment (Writing):
In section 4, it mentions that there is a “fixed-length input sequence of 42 words.” Is this in reference to the sequence of _embeddings_ that are fed to the model? Or is this alluding to how the embeddings are decoded into the expected input?

Author response:
This is the sequence of embeddings that is fed into the encoder of the inversion model.

Reviewer comment (Experiments):
How does the ability to reconstruct text change as a function of the pretrained model used to construct the distribution p(x | v)?

Author response:
We consider this question through many ablations shown in Table 9. Notably, we conclude that we could have seen greatly improved inversion performance if we had trained an inverter with more parameters.

Reviewer comment (Evaluation):
While the results show a significantly better ability to recover prompts than the baseline, the absolute numbers are fairly low – the exact match scores are discouraging and the BLEU scores are also not very high. So I am not convinced if this is a serious concern for security in terms of prompt leakage. For example, looking at the qualitative examples, the inverter struggles with proper names which are often the important target tokens where security vulnerability is concerned.

Author response:
> the absolute numbers are fairly low – the exact match scores are discouraging and the BLEU scores are also not very high. So I am not convinced if this is a serious concern for security in terms of prompt leakage.

We view this work as a proof-of concept, to show that prompts do indeed leak through LM outputs, which had not been known before. Bigger models and more training would likely improve absolute numbers. The work, as is, improves upon the best jailbreaking prompts in most settings.

In addition, we include new experiments that significantly improve both exact match and BLEU score for all out-of-distribution 
settings. Please see the general response for more detail.

Reviewer comment (Evaluation):
Following the two points above, a more pointed analysis on the recoverability of sensitive information instead of recovering general purpose prompts might help establish the significance of the proposed attack more clearly. For example, can the proposed approach differentiate between or recover mildly different prompts that result in similar responses? Conversely, how good is the model at identifying seemingly benign prompt injection attacks? How good is the attack at exploiting specific sensitive information? Overall, I think the attack would be more serious if the exact match numbers are higher or exact match over sensitive tokens (like named entities) is high. Recovering paraphrased general purpose prompts while certainly interesting in its own right, doesn’t seem like a convincing threat.

Author response:
> A more pointed analysis on the recoverability of sensitive information instead of recovering general purpose prompts might help establish the significance of the proposed attack more clearly.

To answer this question, we created a new dataset of prompts that include private data such as names, birthdays, and nationalities. Our models do a good job at reconstructing categorical national information (80% accuracy at countries and 50% on nationalities) while they struggle with exact numbers such as dates. Clearer analysis and examples are available in the new appendix section "Private Prompt Dataset" and the new Table 11.

Reviewer comment (Experiments):
I am not entirely sure about this but the presentation gives me an impression that it only considers “first token” distribution after the prompt. This seems limited – would using token distributions at multiple positions improve the attack?

Author response:
> Would using token distributions at multiple positions improve the attack?

Inverting a prompt using probability outputs from multiple positions is an interesting suggestion. Since this setting would allow us to model the prompt with strictly more information, it would likely improve inversion performance. Although this is easy to test with local models, our threat model is the one represented by standard APIs: a single API call provides probabilities for the next token at a single position.

Reviewer comment (Experiments):
The distribution extraction from access to argmax/top-k logits is very expensive, requiring the number of API calls equal to the size of vocabulary at least. This is likely a concern if multiple positions other than the first token are used.

Author response:
> Distribution extraction from access to argmax/top-k logits is very expensive.

We agree it has some cost and requires lots of calls, but not that it is "very expensive". Extracting the full logit distribution with argmax-only access requires a number of API calls proportional to the vocabulary size V.  For example, in the case of GPT-3.5 Instruct Turbo, using November 2023 pricing, extracting a single prompt can cost as low as $7 (`16 tokens * 3 calls per token * 100k tokens * $.0015 / 1k tokens) = $7.20`). Given the current value of these systems, this cost is not nothing, but it is also not that expensive compared to the engineering cost required to find the most effective prompt for some use cases.