Local Prompt Optimization (2025)

Abstract: In recent years, the use of prompts to guide the output of Large Language Models have increased dramatically. However, even the best of experts struggle to choose the correct words to stitch up a prompt for the desired task. To solve this, LLM driven prompt optimization emerged as an important problem. Existing prompt optimization methods optimize a prompt globally, where in all the prompt tokens have to be optimized over a large vocabulary while solving a complex task. The large optimization space (tokens) leads to insufficient guidance for a better prompt. In this work, we introduce Local Prompt Optimization (LPO) that integrates with any general automatic prompt engineering method. We identify the optimization tokens in a prompt and nudge the LLM to focus only on those tokens in its optimization step. We observe remarkable performance improvements on Math Reasoning (GSM8k and MultiArith) and BIG-bench Hard benchmarks across various automatic prompt engineering methods. Further, we show that LPO converges to the optimal prompt faster than global methods.

## introduction large language models (llms) are everywhere. llms are automating all the tasks that required specialized models a few years ago @xcite . the easiest and cheapest way to control an llm's output is to do prompt engineering @xcite @xcite . unfortunately, writing a prompt is extremely tricky @xcite . although the prompts are in english, the choice of words that effectively have the same meaning makes a huge difference in the prompt's performance on a task @xcite @xcite . furthermore, an llm is inherently biased towards its own vocabulary, making the task even more challenging. thus, llms prompt candidates evaluation prompt proposal best prompt local optimization prompt: let' s think step-by-step edit tokens think : let' s step-by-step identify tokens for optimization prompt: let's step by step. evaluate whether the argument is logically valid by checking if the conclusion follows from the premises traditional global optimization are used to modify prompts in a process called prompt optimization @xcite . prompt optimization techniques follow a twostep process as shown in fig. 1 . first, the prompt is validated against a training set where the incorrect predictions are identified. optionally, a feedback step is added where a natural language feedback, termed 'textual gradients', is obtained by querying the llm @xcite . finally, the prompt is optimized using the textual gradients (incorrect examples or natural language feedback) to obtain an optimized prompt. the cycle is repeated for a fixed number of steps. traditional prompt optimization techniques @xcite @xcite optimize prompts globally ,i.e., mutate all tokens in the prompt. however, optimizing all the tokens in a prompt while searching over the vocabulary to solve a complex problem, makes the prompt optimization very challenging. further, for many production applications, it is desirable to optimize only subsections of the prompt while keeping the other parts static. doing so requires us to limit the scope of the 'prompt proposal' on subsection of the prompt, hence, the need of lo-cal prompt optimization (lpo). thus, we reduce the optimization space (tokens) for the llm to simplify the problem and control the edit direction of a prompt. in this work, we evaluate the efficacy and pitfalls of doing local prompt optimization compared to global prompt optimization. we incorporate local optimization in three automatic prompt optimization algorithms and evaluate on gsm8k @xcite , multiarith @xcite , and big-bench hard @xcite benchmarks. we highlight that local optimization leads to faster convergence of optimal prompt while improving prompt performance. finally, we test local optimization on a real-world application by evaluating it on a production prompt. ## background and method in this section, we will describe a general framework of automatic prompt engineering @xcite and highlight the gap in the framework. building on this, we will introduce local prompt optimization. ## experiments the goal of this section is to highlight the efficacy of local optimization over existing global optimization across different automatic prompt engineering methods. ## conclusion in this work, we identify the gap in the optimization step of the existing automatic prompt engineering techniques. traditionally, prompts are mutated globally in each step. however, this global optimization increases the task complexity as the optimizer (llm) has to work on a larger number of parameters (tokens) to find the optimal up-date. furthermore, many production prompts require optimizing only a section of the prompt and not rewriting the complete prompt from scratch. as a fix, we introduce local prompt optimization (lpo) where we identify the optimization tokens and nudge the optimizer to focus only on those tokens. we observe consistent performance improvements over math reasoning and big-bench hard benchmark. notably, we observe that local optimization searches the optimal prompt significantly quicker than the traditional approach. further, lpo can be integrated well with long prompts, which are more common in practical settings, further showcasing the ubiquity of our method. looking ahead, we are optimistic about prompt optimization techniques built from the perspective of local optimization to benefit from the gains in performance and efficiency.