Position Engineering: Boosting Large Language Models through Positional Information Manipulation (2024)

Abstract: The performance of large language models (LLMs) is significantly influenced by the quality of the prompts provided. In response, researchers have developed enormous prompt engineering strategies aimed at modifying the prompt text to enhance task performance. In this paper, we introduce a novel technique termed position engineering, which offers a more efficient way to guide large language models. Unlike prompt engineering, which requires substantial effort to modify the text provided to LLMs, position engineering merely involves altering the positional information in the prompt without modifying the text itself. We have evaluated position engineering in two widely-used LLM scenarios: retrieval-augmented generation (RAG) and in-context learning (ICL). Our findings show that position engineering substantially improves upon the baseline in both cases. Position engineering thus represents a promising new strategy for exploiting the capabilities of large language models.

## introduction recent advancements in large language models (llms) have demonstrated significant strides towards achieving artificial general intelligence. these models exhibit a wide range of capabilities, such as in-context learning @xcite , answering questions based on documents @xcite , solving complex mathematical problems @xcite , and generating code @xcite . when utilizing llms, user prompts are inputted, converted into sequences of tokens, and then processed through multiple attention layers @xcite . these attention layers employ two types of information derived from the token sequences: (i) semantic information, where the tokens are converted into text embeddings, and (ii) positional information, where the indices of the tokens are converted into positional embeddings @xcite . the attention mechanism then combines the semantic and positional information to predict the distribution of the next token in the sequence. to sentences in prompts. prompt engineering involves either adding, replacing, or removing paragraphs and sentences from prompts. in contrast, the proposed position engineering maintains the original prompt text but incorporates placeholder tokens instead. these placeholders are not involved in the computation of attention scores, thus the computation overhead is not increased. however, they do hold position indices, thereby affecting the position information of other tokens in the text. extensive research has been conducted on modifying prompt text to alter semantic information, aiming to boost task performances. for instance, few-shot prompting is introduced, enabling llms to learn new tasks in an in-context manner @xcite . moreover, the chain-of-thought methodology has been introduced to enhance llms' reasoning abilities by prompting them to produce intermediate tokens @xcite . additionally, automatic prompt engineer has been developed to autonomously design the prompting text for better task-specific performance @xcite . in this study, we investigate the potential of improving performance by solely modifying po-sitional information, without any semantic information change. for the first time, we reveal that downstream task performance can be significantly enhanced by simply adjusting the positional indices of tokens, without modifying the text itself. as illustrated in figure 1 , our approach involves the introduction of placeholder tokens to modify positional information. these placeholder tokens do not contribute to the computation of attention scores; however, they do occupy token indices. consequently, the relative position of other tokens is altered, which could optimize the attention weights among different segments within the prompts. we refer to this approach as position engineering, highlighting the exclusive focus on manipulating positional information. we propose a simple yet effective method based on brutal force to discover the optimal placeholder token number for each downstream task, and experiment it within two prevalent scenarios of llms: retrieval-augmented generation (rag) and in-context learning (icl). our method significantly enhances performance in both tasks, achieving up to a 15.4% absolute increase in accuracy for rag and a 3.6% absolute increase for icl. we also discover that the same placeholder number can consistently improves the rag's performance for different datasets and models. in all, our contributions can be summarized as follows: ## experiments in this section, we present our experiments and findings for position engineering. we evaluate two prevalent tasks for llms, namely retrieval-augmented generation (rag) and in-context learning (icl). our primary testing model is llama2-13b-chat @xcite , although we also expand our experiments to include additional models in the appendix. ## discussion we hypothesize that position engineering serves as a technique to finely adjust the attention weights assigned to different segments within prompts. by extending the positional gap between two segments, the interaction between them is lessened, thereby increasing the attention allocated to other segments. for example, in rag experiments, an increased value of θ a could potentially reduce the impact of the instruction segment while amplifying the attention allocated to the retrieved documents. it is important to note, however, that the initial instruction remains essential, as evidenced in section 3.3. position engineering offers a nuanced approach to adjusting the weights of different blocks without the need for direct addition or removal of text. position engineering offers several advantages: (i) it is easier to optimize due to its numerical search space {θ}, in contrast to prompt engineering, which requires searching over a more complex text space. (ii) it is computationally efficient, as altering position information merely involves updating the position indices input into llms, without increasing the overall computational overhead. (iii) it is orthogonal to prompt engineering, meaning the two approaches can be effectively combined. future works may advance in the following directions. firstly, investigating the internal dynamics of llms can enhance our understanding of position engineering's underlying mechanisms. secondly, employing more sophisticated optimizers, such as gaussian processes or multi-armed bandits, could reduce the search time and discover more refine-grained position editing functions. finally, the exploration of merging position engineering with prompt engineering could harness the full power of llms. ## conclusion in this study, we introduce position engineering, an innovative technique to enhance task performances of llms by merely altering the position information in the prompts. our experimentation with position engineering across a range of tasks and models demonstrates its effectiveness. this approach provides a new avenue for maximizing the capabilities of llms.