A Practical Survey on Zero-Shot Prompt Design for In-Context Learning (2023)

Abstract: The remarkable advancements in large language models (LLMs) have brought about significant improvements in Natural Language Processing(NLP) tasks. This paper presents a comprehensive review of in-context learning techniques, focusing on different types of prompts, including discrete, continuous, few-shot, and zero-shot, and their impact on LLM performance. We explore various approaches to prompt design, such as manual design, optimization algorithms, and evaluation methods, to optimize LLM performance across diverse tasks. Our review covers key research studies in prompt engineering, discussing their methodologies and contributions to the field. We also delve into the challenges faced in evaluating prompt performance, given the absence of a single “best” prompt and the importance of considering multiple metrics. In conclusion, the paper highlights the critical role of prompt design in harnessing the full potential of LLMs and provides insights into the combination of manual design, optimization techniques, and rigorous evaluation for more effective and efficient use of LLMs in various NLP tasks.

## introduction in recent years, transformer-based language models (such as @xcite , @xcite , @xcite , @xcite ) have emerged as a transformative force in the field of artificial intelligence, revolutionizing natural language understanding(nlu) and generation(nlg). as model size and training data have evolved, the gpt series has exhibited extraordinary capabilities in a wide range of natural language tasks by relying on a paradigm known as in-context learning. according to @xcite , in-context learning harnesses the context provided by input data to generate appropriate responses or predictions, contrasting with traditional methods that necessitate explicit task-specific training and fine-tuning on labeled datasets. in-context learning enables large language models to capitalize on vast amounts of data and adapt to various tasks in a flexible and dynamic manner. there are several categories of in-context learning, including zero-shot, one-shot, and few-shot learning. in all types of in-context learning, the key to success lies in effective prompt design, which is occasionally referred to as an "art." this survey paper aims to categorize each type of in-context learning, discuss the core principles, examine state-of-the-art design techniques, and explore recent advancements in in-context learning, with a particular focus on zero-shot discrete incontext learning. ## definition although there is no formal definition for prompt design optimization, we follow the principle from @xcite and provide the definition in (1) for prompt design in in-context learning: here, x i represents input sentences and features, while y i denotes the target labels. θ signifies the parameters for any large language models (llms) or pretrained language models (plms), which remain frozen in the case of in-context learning. f θ represents the output from llms given input x i and prompt p . s is a scoring function that measures the performance of the model output in relation to the ground truth label y i . the objective of in-context learning (or prompt engineering) is to identify the optimal prompt p * that maximizes the score s in the test distribution. based on the structure of p , in-context learning can be further classified into discrete (hard) prompt when p consists of a list of tokens or continuous prompt (soft) where p represents an embedding vector (see figure 1 ). additionally, for zero-shot incontext learning, p is independent of x i , whereas for one-shot or few-shot in-context learning, p can be a function of x i (from training data). this survey focuses on zero-shot in-context learning with discrete prompts and examines its application exclusively in decoder-only llms, such as the gptx series. ## zero-shot discrete prompts with the recent success of large language models such as gpts, designing zero-shot discrete prompts has become increasingly popular in practice. in the experiments conducted by @xcite , the authors demonstrate that carefully engineered zero-shot prompts can actually outperform few-shot prompts. they argue that providing examples does not always help because examples tend to be interpreted as part of a narrative rather than serving as categorical guidance. on the other hand, the advantages of using zeroshot discrete prompts can be listed as follows: (1) zero-shot prompts are highly interpretable, (2) few training data or examples are required, (3) the designing process is more straightforward as we only need to deal with task instructions, and (4) the prompt structure is flexible, allowing us to insert our input wherever needed. zero-shot discrete prompts are also known as task instructions. there are two primary approaches to obtaining a good discrete prompt. the first is heuristic-based manual design, while the second relies on an optimization algorithm to find the optimal prompt. in this section, we focus on reviewing research on prompt design for transformer decoder style models (e.g., gpt), which has been the focus of a majority of research efforts. ## conclusion the rapid development of large language models (llms) has significantly influenced various nlp tasks. among the techniques to harness their capabilities, in-context learning with different types of prompts-discrete, continuous, few-shot, and zero-shot-has shown remarkable promise. discrete prompt engineering emphasizes humanreadable prompts that can enhance model performance, while continuous prompt optimization involves soft prompts that can be learned and opti-mized directly in the same language model. fewshot learning leverages a small number of examples to guide the model in the right direction, whereas zero-shot discrete prompts only require task instructions, offering a more straightforward design process. manual design of prompts can be guided by principles based on model behavior, and optimization algorithms can be used to find optimal prompts. evaluating the performance of prompts is challenging, as there is no single "best" prompt, and various metrics need to be considered. in conclusion, as llms continue to evolve, prompt design remains a crucial factor in harnessing their full potential across a wide range of applications. a combination of manual design, optimization techniques, and rigorous evaluation can lead to more effective and efficient use of llms in diverse nlp tasks.