Revisiting Automated Prompting: Are We Actually Doing Better? (2023)

Abstract: Current literature demonstrates that Large Language Models (LLMs) are great few-shot learners, and prompting significantly increases their performance on a range of downstream tasks in a few-shot learning setting. An attempt to automate human-led prompting followed, with some progress achieved. In particular, subsequent work demonstrates that automation can outperform fine-tuning in certain K-shot learning scenarios. In this paper, we revisit techniques for automated prompting on six different downstream tasks and a larger range of K-shot learning settings. We find that automated prompting does not consistently outperform simple manual prompting. Our work suggests that, in addition to fine-tuning, manual prompting should be used as a baseline in this line of research.

## introduction transformer-based large language models (llms) are now considered foundation models for downstream tasks @xcite . the pre-train then fine-tune approach achieved state-of-the-art performance on a range of natural language processing (nlp) tasks @xcite @xcite . unfortunately, in many nlp applications, the lack of high-quality labelled training data is a barrier to producing a model with good performance in the pre-train and then fine-tune approach. to address this issue, prompt-based learning @xcite @xcite emerged as a new paradigm for tuning a high-quality, pre-trained llm in a few-shot learning scenario, where only a few samples are available for downstream task learning. in the prompt-based learning paradigm (figure 1 ), an input x is modified using a template function p, also known as a prompting function and has one or more placeholders called mask tokens i t @xmath0. a gor geous, wi t t y, seduct i ve movi e. bad good gr eat . . . 0 1 i t @xmath1. cont ai ns no wi t , onl y l abor ed gags. figure 1 : sentiment analysis with the prompt-based learning paradigm. input x ′ is the prompted input, and there is a many-to-one mapping between answers @xmath2. prompting formulates downstream tasks such as sentiment analysis and text classification to cloze completion (also known as filling in the blanks). furthermore, using prompts and fine-tuning allows models to gain superior few-shot learning capabilities @xcite @xcite . despite the relative success of prompt-based learning, the design of prompts can be a challenging task. as a result, many research studies sought to automate the process of designing suitable prompts for downstream tasks @xcite @xcite . the motivation for automating prompt design is usually two-fold: first, manually designing prompts can be time-consuming; and second, automated ones can often provide better performance. in this work, we question the second motivation and demonstrate that existing automated prompts do not consistently outperform their manual counterparts under various k-shot learning setups. in this paper, we make the following contributions: ## related work the rise of the prompting-based learning paradigm comes with the development of llms @xcite , which were demonstrated to be good few-shot learners @xcite . to begin with, researchers focused on manually crafted prompts for downstream tasks @xcite @xcite @xcite ), yet soon shifted towards automated prompt designs. schick et al. investigated how to automatically identify label words for a prompt (schick and schütze, 2020a,b), while shin et al. proposed autoprompt, a framework for automatically generating prompts for various tasks, through a gradient-based search (shin et al., 2020). gao et al. used another llm, t5 @xcite , to generate both the prompting templates and verbaliser answer domains @xcite . han et al. incorporated logic rules into prompt designs, combining several simple subprompts according to these rules @xcite . all of the above mentioned methods are based on the assumption that the prompt design has to rely on discrete tokens. @xcite demonstrated that prompts could be trainable continuous embeddings, or soft prompts, instead of discrete tokens. these soft prompts can be learned with a frozen language model (llm) on a target task @xcite @xcite . liu et al. further discovered that deep prompts, which are soft prompts used in every layer of the model, allow for scaling to large llms for complex natural language processing (nlp) tasks @xcite . zhang et al. developed differentiable prompts, which put the label tokens design of the prompt into a continuous space and optimised it jointly with soft prompts @xcite ). an extensive evaluation was conducted by zhang et al. on various downstream tasks. most of the work on automating prompt design mentioned above has two major motivations: to reduce the amount of time it takes to design prompts manually; and to potentially gain better performance, since manual prompt formats can be sub-optimal @xcite . while the first motivation may be valid in some cases, it largely depends on the task complexity and the amount of data available -it is sometimes possible for nonexperts to design a prompt sufficient for simple tasks with a large amount of data. the principal focus of this work, however, is on the second motivation: can automated prompts really outperform manual prompts in a consistent manner? a comparison between automated and manual prompts is lacking in current research. to our knowledge, automated prompting methods focus solely on comparing to fine-tuning in a few-shot learning setup, while a comparisons to manual prompting methods remain unexplored. in this paper, we consider au-toprompt (auto) @xcite and differential prompt (diff) @xcite as representatives, where one is based on discrete tokens, while the other is based on continuous embeddings. we compare them with manually designed prompts and fine-tuning without prompting on various tasks. ## conclusion in this paper, we revisit the results generated from automated prompting, and show that automated prompting cannot consistently outperform simple manual prompting on a variety of tasks. we also demonstrate that the performance of automated prompting is heavily dependent on the amount of data available, and in some cases can even be worse than fine-tuning. on the other hand, manual prompting is more robust to the amount of data available, and can have similar performance to finetuning if not outperforming. we take a closer look at the prompts and verbalisers generated by automated discrete prompting (autoprompt) and point out that few-shot learning settings make it challenging to generate prompts and verbalisers that perform well. we hope that this work will motivate researchers to use manual prompts as a general baseline.