Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System (2023)

Abstract: Neural language models are increasingly deployed into APIs and websites that allow a user to pass in a prompt and receive generated text. Many of these systems do not reveal generation parameters. In this paper, we present methods to reverse-engineer the decoding method used to generate text (i.e., top-_k_ or nucleus sampling). Our ability to discover which decoding strategy was used has implications for detecting generated text. Additionally, the process of discovering the decoding strategy can reveal biases caused by selecting decoding settings which severely truncate a model’s predicted distributions. We perform our attack on several families of open-source language models, as well as on production systems (e.g., ChatGPT).

## introduction language models are increasingly being incorporated into web applications and other user-facing tools. 1 these applications typically do not provide direct access to the underlying language model or the decoding configuration used for generation. in this paper, we show how even in this blackbox setting, it is possible to identify the decoding strategy employed for generation. we consider the case where one only has access to a system that inputs a prompt and outputs a generated response. we present algorithms for distinguishing the two most popular decoding strategies, top-k and nucleus sampling (a.k.a. top-p), and estimating their respective hyperparameters (k and p). the choice of decoding strategy-the algorithm used to sample text from a language model-has a profound impact on the randomness of generated text, introducing biases toward some word choices. for example, when openai's chatgpt, 2 a chatbot built with large language models, is repeatedly passed a prompt asking it to report the outcome of rolling a twentysided die, we found that it only returns 14 of the 20 options, even though all should be equally likely. prior work has shown that knowing the decoding method makes it easier to detect whether a writing sample was generated by a language model or else was human-written @xcite . as generated text proliferates on the web, in student homework, and elsewhere, this disambiguation is becoming increasingly important. concurrent work to ours by @xcite has developed similar strategies for detecting decoding strategy from a blackbox api: however, they focus more on identifying hybrid decoding strategies (including beam search), whereas we focus more on prompt engineering to produce close-to-uniform token distributions that reduce the number of queries needed. our proposed methods complement but are not comparable to those of @xcite . their method trains classifiers that input a generated text sequence and output a prediction for the decoding strategy used to generate it. in contrast, our method interacts with an api and does not require any data or ml training. ## background neural language models are not inherently generative. a causal language model f θ takes as input a sequence of tokens x 1 ,...,x t-1 and outputs a score for each possible next token x t , computing the a likelihood score for each token in the vocabulary, which can be transformed into a probability distribution by applying a softmax such that prob(x t |x 1 ,...,x t-1 )∼f θ (x 1 ,...,x t-1 ). a decoding method takes this probability distribution as input and samples a particular token to output. the simplest algorithm is argmax decoding (also known as 'greedy decoding'), where the most likely next token is outputted. argmax is rarely used in practice because (1) only one generation can be produced for any given prompt, and (2) generations with argmax tend to be repetitive and low-quality. most commonly used decoding algorithms are based on random sampling: a token is chosen with probability proportional to the likelihood assigned to it by the model. whereas argmax sampling has too little randomness, purely random sampling over the full distribution can have too much, leading to text that is too erratic and prone to errors. thus, it is common to modify the distribution to reduce entropy before sampling from it. in this short paper, we focus on two popular strategies researchers have developed for decoding: top-k sampling @xcite and top-p sampling @xcite (also known as nucleus sampling). top-k sampling involves the implementer picking a fixed hyperparemter k then only ever sampling from the k most likely items by assigning all other items a score of 0 before applying the softmax. top-p sampling involves the implementer picking a fixed hyperparamter p. then at each step t of generation, a k t is selected such that the k t most likely vocabulary items cover p proportion of the total probability mass in the distribution. more precisely, let the notation x (l) refer to the lth most likely token in the distribution predicted at step t. we set k t to the first value for which kt @xmath0,...,x t-1 ) ≥ p. then, the distribution is truncated to the k t most likely tokens, as described above for top-k. other common methods like beam search and temperature annealing are omitted in the interest of space (cf. @xcite and @xcite ). temperature annealing simply modifies the probability distributions of the output tokens, so the methods in this manuscript can be easily generalized (and indeed were in the concurrent work of @xcite ). beam search is a bit more complicated, as tokens are not chosen independently of previous tokens; instead, multiple candidate token paths are retained. as such, it would be necessary to generate more than a single word for each prompt, which is the primary interrogative tool we use here. ## experiments we conduct experiments on four language models where we can set the decoding strategy: gpt-2 base and xl @xcite , gpt-3 davinci @xcite , bloom 3b (scao et al., 2022), and pythia 2.7b @xcite . ## limitations our method is limited to identifying when top-p sampling or top-k sampling is used. we do not attempt to detect other decoding strategies which other systems might use. additionally, there is no guarantee that a system would use a single decoding strategy-it is possible that different prompts may trigger different decoding strategies, or that a/b testing results in different users seeing different decoding strategies. our chatgpt results were computed by two different authors on separate openai accounts. also, we have no guarantees that the decoding strategy is not changed over time. some of our chatgpt results were computed using the december 15, 2022 release while others were computed using the january 9, 2023 release. additionally, the biases in distributions that we see here could have other underlying reasons; for example, changes in the data can result in very different true distributions. furthermore, under the hood, an api might not be generating a new random generation each time an identical prompt is passed in. either random seeds might be getting re-used, or generations could be retrieved from a cache. in both cases, the generations might look like argmax sampling. it's also conceivable that certain combinations of fixed models could look like top-k/p. for example, if a query is randomly routed to one of a series of s servers, each serving a different model, we might interpret the decoding strategy to be top-k even if each server is using argmax. in these cases, an approach more like that proposed by @xcite , where classification of decoding strategy is made based on a long generated sequence (rather than single token system predictions, as in our approach), might be more effective. for top-p (though not top-k), we require access to an underlying distribution that approximates the model used. this is not an issue for open source models or models with api access that allows specifying the decoding strategy, but it does limit the applicability of our method to newer proprietary models. it may be possible to empirically determine distributions for carefully engineered prompts, but future work is needed for reverse engineering fully closed models. ## conclusion our attack shows that with even a little work, it is possible to reverse-engineer common decoding strategies. although we have focused here only on top-p and top-k sampling, these approaches generalize readily to other common methodologies when the output probability distributions are well-approximated. along with other recent work on reverse-engineering other parts of a language generation system @xcite , it seems is infeasible to hide inference implementation details given black-box access to the system.