Unnatural language processing: How do language models handle machine-generated prompts? (2023)

Abstract: Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are routinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vectors from a model’s embedding space. We use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. We study the behavior of models of different sizes in multiple semantic tasks in response to both continuous and discrete machine-generated prompts, and compare it to the behavior in response to human-generated natural-language prompts. Even when producing a similar output, machine-generated and human prompts trigger different response patterns through the network processing pathways, including different perplexities, different attention and output entropy distributions, and different unit activation profiles. We provide preliminary insight into the nature of the units activated by different prompt types, suggesting that only natural language prompts recruit a genuinely linguistic circuit.

## introduction neural language models (lms) are parameterized probabilistic models that can assign a probability to any sequence of language tokens. given that they are trained on huge amounts of natural language, we expect their statistics to mimic those of the latter. in this paper, we study what happens when a lm trained on english must process "unnatural language", that is, sequences that are extremely unlikely in english, as they are syntactically and semantically ill-formed. we tackle this topic through the lens of machinegenerated prompts, that is, automatically discovered input token sequences that optimize the model's performance in a target zero-shot task @xcite . it has indeed been widely observed that such prompts, while empirically effective, consist of nonsensical sequences of jumbled tokens. for example, using the popular autoprompt algorithm of @xcite and the opt-1.3b language model @xcite , we found that the prompt "[x] @xmath0. even more extremely, recent prompt generation methods find sequences of embedding vectors that do not correspond to items in the model vocabulary, but still outperform both human-crafted and machinederived discrete prompts @xcite @xcite . this state of affairs is paradoxical: why does a lm that has been trained to reproduce the statistics of natural language respond better to input sequences that are completely outside this distribution? we present a detailed comparative study of how lms internally process manually-crafted prompts and both discrete and continuous machinegenerated prompts. while we do not solve the puzzle of why linguistically ill-formed machinegenerated prompts are better than human prompts, we discover that there are fairly deep differences characterizing the various prompt types through all the processing stages of a lm, suggesting that the latter has fortuitously developed a distinct pathway to process unnatural language. understanding lms more generally, understanding how lms process unnatural linguistic input contributes to our understanding of their inner workings. therefore, our study is also related to work on interpretability (lipton, 2018), defined as the analysis of a trained model's decision policy. in particular, one can approach neural network interpretability by adopting a mechanistic paradigm, consisting in directly studying the weights and their activation in order to reverse-engineer the neural network. successful mechanistic insights have been obtained in computer vision @xcite . @xcite is an example of mechanistic interpretability applied to tranformer lms. in this context, the transformer feed-forward layers have been shown to behave like key-value memories @xcite . notably, as shown in @xcite , these memory slots, also called knowledge neurons, encode specific concepts acquired during pre-training. even more interesting, manually editing these memories allows to causally control the prediction output @xcite , suggesting that they play a central role in language processing (see also @xcite . in the present paper, we show that unnatural language processing is achieved by recruiting different knowledge neurons than the ones used for natural language processing. ## processing machine-generated prompts we experimentally demonstrate that differences between human and machine-generated prompts exist at three different levels: (1) at the input level, when comparing prompt types in the embedding space, (2) at the output level, when analyzing predictions and output probabilities, and (3) at the level of intermediate activation, indicating a difference in processing at work in the lm. we conclude this quantitative analysis by showing that, although these metrics are correlated when compared within the same prompt type, the correlation is weak between prompts of different types, leading to a number of counterintuitive patterns in lm prompt processing. high accuracy and high perplexity as confirmed in ## discussion we have studied the phenomenon of linguistically and semantically opaque machine-generated prompts from the perspective of how lms process them, compared to human-crafted ones. our study has important limitations, that are discussed in the relevant section below. however, at least for the prompt generation methods, lms and tasks we explored, we can draw some general conclusions. more than a "happy accident" our evidence suggests that the differences between human and machine-generated prompts are not just superficial, but affect all levels of network processing, and result in the activations of qualitatively different units. some of these units are stable across semantic tasks, suggesting that they are more generally recruited to process any "unnatural" input. moreover, contrary to what one could reasonably predict, there is some evidence that machine prompts are more robust than human ones, in the sense that they achieve better output calibration. it's unlikely that the lm has been exposed to anything like m-disc prompts during its initial training, and definitely it could not have seen out-ofvocabulary m-cont prompts, so we can only assume that the special pathways triggered by these prompts arose through unforeseen side effects of pre-training. foot_8 however, they seem to be more than just lucky connectivity accidents exploited by specific prompts to solve specific tasks, or else it would be difficult to explain the overall low entropy of machine prompt predictions and the commonalities in the units they activate. moreover, there is evidence that m-disc prompts can transfer across transformer-based lms @xcite , suggesting that unnatural language pathways might arise from the interaction of general characteristics of the transformer architecture with web-derived training data that are partially shared across many current pre-trained lms. we must defer a better understanding of the nature of these unnatural pathways to future work. in particular, we plan to zoom further in into the processing of specific templates, tracking their processing throughout the network with methods such as the vocabulary-based unit analysis of section 5.