Learning to Instruct: Fine-Tuning a Task-Aware Instruction Optimizer for Black-Box LLM s (2025)

Abstract: The performance of Large Language Models (LLMs) critically depends on designing effective instructions, which is particularly challenging for black-box LLMs with inaccessible internal states. To this end, we introduce Learning to Instruct , a novel paradigm that formulates instruction optimization as an LLM fine-tuning objective for a white-box “instruction engineer” LLM, leveraging its rich learning capacity and vast pre-trained knowledge to enable efficient and effective instruction optimization. Within this paradigm, we propose Automatic Instruction Optimizer (AIO), a novel framework that fine-tunes a white-box LLM into a capable instruction engineer. AIO learns to optimize task-aware, human-comprehensible instructions by incorporating task nuances and feedback from the task-solving black-box LLM. To overcome the challenges of inaccessible black-box gradients and high API costs, AIO introduces a novel zeroth-order (ZO) gradient approximation mechanism guided by Thompson Sampling (TS), which reuses informative black-box LLM feedback for improved query efficiency. Extensive experiments show that AIO generally outperforms strong baselines in both effectiveness and efficiency, establishing Learning to Instruct as a promising new direction for black-box LLM instruction optimization.

## introduction large language models (llms) have shown remarkable capabilities. however, their strong performance generally relies on high-quality instructions that provide clear guidance and relevant context. crafting such instructions is particularly challenging for powerful black-box (api-based) llms @xcite , due to their inaccessible internal states. meanwhile, optimal instruction strategies can vary significantly across different llms, model versions, and downstream tasks @xcite @xcite . manual instruction crafting, while sometimes effective, can be costly, labor-intensive, and scales poorly with increasing task complexity @xcite @xcite , highlighting the need for automated and task-aware mechanisms to optimize black-box llm instructions @xcite . formally, instruction optimization can be viewed as a special case of adapting powerful black-box models to specific downstream tasks, by identifying effective task-specific instructions. to this end, a line of existing research casts a white-box llm into a capable "instruction engineer" to instruct the black-box model through in-context learning frameworks @xcite @xcite . on the other hand, fine-tuning offers a mechanism for integral adaptation of the llm. the capacity to adjust model parameters allows for a deeper assimilation of taskspecific nuances and complex patterns @xcite @xcite . such fine-grained adaptability is particularly important for casting the "instruction engineer" llm, where accurately capturing and conveying task requirements is crucial for generating high-quality instructions, representing a promising yet underexplored direction of instruction optimization. motivated by these insights, we propose learning to instruct, a novel paradigm that re-frames instruction optimization as a fine-tuning objective for a white-box "instruction engineer" llm. this formulation helps unlock the full potential of the optimizer by enabling parameter updates through methods like parameter-efficient fine-tuning (peft) @xcite , allowing it to effectively and efficiently leverage pre-trained knowledge to capture subtle task nuances. this conceptual shift from in-context learning to a fine-tuning objective is our first contribution, offering a principled path to instruction optimization. however, applying the learning to instruct paradigm to black-box llms presents a significant technical hurdle: the inaccessible internal gradients of the task-solving llm. this prevents direct, gradient-based fine-tuning of the white-box instruction engineer. while zeroth-order (zo) approximation methods can bypass the need for direct gradients @xcite , their reliance on random sampling can lead to potentially prohibitive query complexity, making them impractical given the generally expensive api costs of powerful black-box models. to overcome this challenge and make our paradigm practical, we introduce the automatic instruction optimizer (aio) framework, centered on a novel and query-efficient zo fine-tuning pipeline. our approach tackles the potentially high cost of zo approximation through two synergistic advances. first, we identify that the zo approximation difficulty (e.g., slower convergence) increases with the number of parameters, i.e., the dimensionality of the gradients to be estimated @xcite . in light of this, aio proposes to reduce the cost by estimating gradients only with respect to the output logits of the white-box llm (which generate the instruction), rather than its entire parameter set. this consequently lowers the gradient estimation dimensionality, thus reducing the number of queries required. second, to maximize the information gained from each query, aio reformulates the selection of perturbation directions as a contextual bandit problem. it employs a thompson sampling (ts)-based strategy to intelligently reuse feedback from past queries, prioritizing directions most likely to improve the instruction. this ts-aided zo method is our second core contribution, providing a feasible and efficient mechanism to realize the learning to instruct paradigm for black-box llms. next, we review related works in sec. 2 and define problem formulation in sec. 3. sec. 4 details the aio framework with ts-aided zo method. experiments showcasing aio's effectiveness and efficiency are in sec. 5, and we conclude in sec. 6. llm-based instruction generation. llmbased instruction optimization is an emerging research topic @xcite @xcite , where llms are applied as instruction optimizer and their instructing strategies are gradually refined based on target model feedback. in particular, there are a series of works leveraging meta-prompts, which can be manually designed by humans @xcite , or optimized by llms @xcite . meanwhile, @xcite perform in-context "gradient descent" on instructions based on interactions with an "instruction engineer" llm. @xcite ; @xcite propose in-context evolutionary algorithms to refine llm-generated instructions. aio alternatively leverages rich learning capacity of fine-tuned llms to model the relationship between task information and black-box llm feedback, without relying on human experts. ## problem formulation under the proposed learning to instruct paradigm, given a target task t , we involve two llms: (1) a black-box llm f b (•) is for task-solving, i.e., generating answers for task queries. the black-box llm is treated as part of learning objective, as we aim to optimize instructions that enhance its performance. (2) a white-box llm f w (•; θ w ) with trainable parameters θ w will generate and refine human-comprehensible instructions, with information from task t and feedback from f b @xcite . task data and instruction generation. suppose the target task t is associated with three separate data collections (i.e., query-answer pairs (x, y )): (1) training data (i.e., task exemplars) d train guides the white-box llm in generating and optimizing task-specific instructions; (2) validation data d valid is used for performance evaluation during optimization; (3) final evaluation is conducted on a separate test set. let f w (d train ; θ w ) learning objective. given exemplars d train and an evaluation function l(•, •), instruction optimization is framed as a fine-tuning objective. we aim to find the optimal white-box llm parameters θ w that optimizes task-specific score. with a shorthand ϕ(θ w ) := f w (d train ; θ w ) for generated instruction. intuitively, we utilize finetuning to guide how the white-box llm comprehends task exemplars and composes task-specific instructions, based on task-solver black-box llm feedback during optimization. ## automatic instruction optimizer (aio) given our learning objective (eq. 1), a straightforward approach is to update the white-box llm parameters θ w , using gradients computed from instruction evaluation results. however, the nested black-box llm makes direct back-propagation towards θ w via the chain rule infeasible. validation score. given our optimization objective in eq. 1, since a comprehensive overview of the task distribution t can be inaccessible, we evaluate the performance of the generated instruction on the validation data d valid , as per validation score ## experiments for our experiments, llama-3-8b-instruct @xcite complementary experiments in appendix c. ## conclusion in this paper, we introduce learning to instruct paradigm for black-box llm instruction optimization, which formulates the instruction optimization process as an llm fine-tuning objective, to harness the representation power of fine-tuned llms. we also propose aio, a novel framework for optimizing task-specific black-box llm instructions. by fine-tuning a white-box llm as an instruction optimizer, aio learns from task nuances and black-box llm feedback to achieve adaptive instruction optimization. given inaccessible black-box llm gradients, we introduce a novel ts-aided zeroth-order gradient approximation method, for tractable and efficient white-box llm fine-tuning. extensive experiments demonstrate aio's superiority in both performance and api token efficiency, with additional analysis highlighting its properties. future extensions of aio are discussed in appendix f.