Do Prompt Positions Really Matter? (2024)

Abstract: Prompt-based models have gathered a lot of attention from researchers due to their remarkable advancements in the fields of zero-shot and few-shot learning. Developing an effective prompt template plays a critical role. However, prior studies have mainly focused on prompt vocabulary searching or embedding initialization within a predefined template with the prompt position fixed. In this empirical study, we conduct the most comprehensive analysis to date of prompt position for diverse Natural Language Processing (NLP) tasks. Our findings quantify the substantial impact prompt position has on model performance. We observe that the prompt positions used in prior studies are often sub-optimal, and this observation is consistent even in widely used instruction-tuned models. These findings suggest prompt position optimisation as a valuable research direction to augment prompt engineering methodologies and prompt position-aware instruction tuning as a potential way to build more robust models in the future.

## introduction recently, @xcite have shown the impressive performance of using handcrafted prompts with a frozen language model in zero-shot and fewshot learning, leading to increased interest and activity in prompt engineering within the nlp community (schick and schütze, 2020; @xcite . prompting (a.k.a promptbased learning @xcite ) aims to reformat an nlp problem so that it closely matches the format used in the pre-training tasks. to apply prompt-based learning methods effectively, a critical step involves the creation of a prompt template that maximizes performance on the downstream task. in many previous works, it is common to manually pre-define a template while keeping the prompt * corresponding author position fixed (e.g. prepend the prompt to the input @xcite ). these studies often concentrate more on either prompt vocabulary searching @xcite @xcite or prompt embedding initialization @xcite . however, there has been limited research exploring how different approaches to positioning the prompt sequences can affect the models' behaviour, despite indications that varying prompt positions may lead to performance difference @xcite . hence, in this paper, we quantify how much prompt positions matter by evaluating various accessible models on different nlp tasks under fewshot and zero-shot settings. we comprehensively test a range of prompt position options with many widely used prompt styles (e.g. cloze and prefix) and methods (e.g. gradient-based and gradientfree). our findings reveal unexpected performance variations among different prompt positions in both zero-shot and few-shot settings. we also discover that instruction-tuned models do not always reduce performance disparities, even though they typically include vocabulary and positional variations in their training templates. interestingly, we observe that in many cases, the prompt positions used in previously published works show a sub-optimal performance compared to other prompt position choices. our choice of zero and few-shot tasks is driven by the observation that prompting methods are particularly useful when training data is limited @xcite , and this hypothesis is born out by our results which show prompt positions matter more when the available labelled data is limited. the key contributions of this paper are foot_0 : ## related work prompt-based learning. many prior works have concentrated on gradient-based methods within discrete spaces (schick and schütze, 2020; schick and schütze, 2020; @xcite as well as prompting directly in the embedding space. this latter approach uses tunable prompt tokens that are not limited to natural language, which can be either prepended to the input @xcite @xcite or be inserted in a hybrid template @xcite . @xcite optimize continuous tokens without using gradients, although this approach is not suitable for apis like gpt-3 which only allow for text modifications rather than token embeddings. there are also gradient-free works focusing on in-context learning @xcite , chainof-thought @xcite @xcite , and instruction generation @xcite , especially when instruction tuning @xcite @xcite plays a key role in the steering process of large pre-trained lms. our paper includes experiments examining the impact of prompt position from both gradient-based and gradient-free perspectives. prompt position. there is limited work that involves prompt position. @xcite find that the position of a handcrafted prompt (before or after the input) affects model performance, but there's no consensus on which position is best. @xcite propose an instance-dependent prompt generation method; meanwhile, they study the effect of inserting a sequence of prompt tokens in different positions based on their proposed method and prompt tuning @xcite . @xcite have proposed a dynamic position method that can significantly improve the performance of prompt tuning. they both point out that different positions of prompts will deliver different results with the consideration of only one specific approach to creating the prompt. in this paper, we present the most comprehensive analysis of prompt positions to date and take into account various types of prompts under both zero-shot and few-shot settings. ## general discussion our main research question in this paper is whether and how prompt positions matter. in section 4, we observe that continuous prompts are more sensitive to prompt positions compared to manual prompts. in section 5, all language models exhibit a certain degree of sensitivity to positional changing in both few-shot and zero-shot settings. both sections 4 and 5 highlight that the optimal prompt position is not shared across tasks, and sometimes even differs among items of data. furthermore, section 5 reveals that instruction-tuning fails to mitigate positional variance, possibly due to an unequal distribution of position variety in instruction templates. our goal is not to identify a 'best' position to replace the prompt position used in prior works. rather, we aim to highlight the effect of prompt position, which is often overlooked before. for gradient-based prompting such as prompt tuning, due to the influence of training data items, we suggest instance-dependent prompt position optimisation as a valuable direction, with the potential to enhance model performance, which is similar to @xcite ; yang et al. @xcite . as for gradient-free prompting, in real-world scenarios, prompt positioning and the logic in which questions or instructions are structured are diverse. a robust language model should display a consistent understanding of inputs with the same semantic essence when only the prompt position changes. a future work direction is to explore if increasing the position variety during the instruction tuning process improves the robustness of pre-trained models. ## conclusion in this study, we evaluate the effect of prompt position in both zero-shot and few-shot scenarios. our findings reveal significant variations in performance among different prompt positions across a variety of methods, tasks, and models. additionally, our research indicates that prompt positions commonly adopted in the existing literature often result in sub-optimal performance, with no single prompt position universally excelling across all tasks. these findings suggest prompt position optimisation as a promising new direction in prompt engineering and advocate for the consideration of position-aware instruction tuning to develop more robust models in the future.