Strings from the Library of B abel: Random Sampling as a Strong Baseline for Prompt Optimisation (2024)

Abstract: Recent prompt optimisation approaches use the generative nature of language models to produce prompts – even rivaling the performance of human-curated prompts. In this paper, we demonstrate that randomly sampling tokens from the model vocabulary as “separators” can be as effective as language models for prompt-style text classification. Our experiments show that random separators are competitive baselines, having less than a 1% difference compared to previous self-optimisation methods and showing a 12% average relative improvement over strong human baselines across nine text classification tasks and eight language models. We further analyse this phenomenon in detail using three different random generation strategies, establishing that the language space is rich with potentially good separators, with a greater than 40% average chance that a randomly drawn separator performs better than human-curated separators. These observations challenge the common assumption that an effective prompt should be human readable or task relevant and establish a strong baseline for prompt optimisation research.

## introduction pre-trained large language models (plms, @xcite @xcite @xcite have demonstrated remarkable performance when conditioned with appropriate context @xcite @xcite @xcite . for instance, when given a query along with the phrase "let's think step by step," such models are capable of solving reasoning tasks @xcite . these special tokens, often called "separators", are usually placed at the end of the input data or at the beginning of the output ( the sentiment is ... horse battery staple ... unlike relying on human knowledge or using external large language models to suggest alternatives, we find that randomly selected separators from the vocabulary can also yield good performance. recent work found that including thinking-style separators such as "take a deep breath" can significantly enhance reasoning performance @xcite . similarly, another study discovered that simply using "solution:" is even more effective @xcite . these findings suggest that the language space for separators might still be under-explored, with many effective options yet to be identified. a common framework employed in this line of research involves starting with thinking-style separators, using a language model to generate alternatives, and then selecting effective separators based on certain criteria @xcite @xcite . this optimisation-style framework has shown promise in automating the exploration of the language space, which addresses the challenge of relying on human expertise to develop task-specific separators. most existing methods, particularly those applied to reasoning tasks, assume that effective separators should be closely related to the task or con-text @xcite @xcite . however, perhaps counter-intuitively, we find that a performant separator does not necessarily have to be task-relevant or even coherent. sometimes, even tokens chosen at random from the vocabulary can improve performance as much as semantically meaningful phrases. ## random separator optimisation we propose a random separator optimisation framework (figure 2 ) to find effective separators based on random sampling. the framework consists of three components: 1) random separator generation, 2) separator evaluation, and 3) separator selection. ## results and discussion we report our results in ## related work automatically discovering effective prompts remains a challenging research problem because of the complexity of the search space. one direction is continuous prompt tuning @xcite @xcite , which involves adding a set of smaller tunable parameters to pretrained language models. an alternative approach is to optimise discrete token spaces. @xcite showed that gradient information at the embedding layer can guide the discovery of more effective prompts. according to their research, unnatural prompts can also result in good performance, which matches our observations. in spite of the efficiency of using gradient information, such a method, which heavily relies on the availability of language models, imposes some restrictions on certain types of models. an alternative direction is black box search. to simplify language space optimisation, @xcite introduced a set of operations, such as add/delete/replace tokens. ape @xcite showed that generating some alternatives and then selecting and rephrasing them could also provide effective solutions. similarly, @xcite used evolutionary methods to optimise the search process. recently, @xcite demonstrated that we can teach language models to learn the pattern of good prompts using human-written meta prompts. evoprompt @xcite showed how we can formulate the evolutionary process in meta-prompt. @xcite further demonstrated how we can improve the meta-prompt using language models, making the whole framework completely automatic without relying on the internal state of language models. ## conclusion we find that random separators, even those selected at random from vocabulary, could be as effective as previously discovered state-of-the-art prompts. in addition, we conduct research on three different types of random separators, which demonstrated that these random separators do not require instruction-tuned models, could provide a 12% relative improvement as compared to human baselines, and are on par with a self-optimising approach involving complex meta-prompt engineering.