L ang P ro B e: a Language Program Benchmark (2025)

Abstract: Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost-quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance.

## introduction language models are now routinely used to build modular natural-language software systems that process data or serve bespoke applications. previous work @xcite @xcite refer to such systems as language programs 1 , where language model calls are wrapped into highly structured modules and pipelined with external tool calls. language programs often compose betterscoped lm capabilities @xcite , offer structure for models to access tools or information @xcite @xcite , and systematically scale figure 1: langprobe includes 15 datasets, 4 optimizers, and more than 10 language programs, creating more than 400 configurations for evaluating language programs with different tasks and optimizers. to support such programming and permit portability across models and tasks, recent work has introduced declarative languages for expressing these systems and automating prompting (or finetuning) for their modules. for example, given a task-specific objective, frameworks like dspy @xcite and textgrad @xcite offer optimizers for language programs. by composing techniques like bootstrapping few-shot examples, refining free-form instructions, or fine-tuning, the optimizers transform the baseline language program into a new, optimized language program with updated parameters (including both lm prompts and lm weights), which often align better with a distribution of inputs or a nuanced task. despite the interest in this space, it remains unclear which problems actually need modular programs, especially as models continue to improve, or which types of architectures and optimizers will work best for different problems. to study this, we introduce langprobe (language program benchmark) (figure 1 ), a benchmark for evaluating combinations of language models, program architectures, and their optimizers. for many datasets, langprobe implements multiple language programs (section 3.2), ranging from a single lm call to sophisticated and modular systems. using langprobe, we run over 2000 experiments to investigate several research questions. do language programs and optimizers for them improve cost-performance compared to using raw model calls? (section 5) on average, we find that language programs show non-trivial improvement over raw model prediction baselines even while considering language model costs. for example, the best optimized program running on gpt-4o-mini performs better than both the gpt-4o and o1-mini raw model prediction baselines at a significantly cheaper cost. however, this is far from uniform, as many self-contained problems (e.g., mmlu) can be straightforwardly tackled by powerful models without composition or optimization. what language program architectures work best on different problems? (section 6) we find that modular programs are, perhaps expectedly, indispensable for tasks whose specification demands or strongly encourages access to external information or other tools. for example, tasks that require composing long-tail world knowledge benefit from retrieval-augmented generation (rag) programs and multi-hop retrieval. while inference-time scaling programs are known to be helpful in certain cases, we find that they can fail to exceed baseline systems in certain applications, e.g. when error compounds across different modules. this illustrates a general theme we find, in which human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. which optimizers perform best and where? (section 7) all optimizers can provide quality gains for many-though not all-combinations of models, tasks, and programs, but different optimizers create rich tradeoffs in terms of optimization and inference costs. on average, we find that mipro @xcite , an optimizer that constructs and explores combinations of instructions and few-shot examples through bayesian search, performs best overall. as in prior work, we find that searching over combinations of selfbootstrapped few-shot examples (bootstrapfew-shotrandomsearch) is a highly competitive simple strategy, though the benefits of (bootstrapped) demonstrations heavily depend on the model-task pair. while this lies outside our scope, we believe that langprobe defines a general methodology and large headroom for new prompt optimizers and also for finetuning-or rl-based optimizers to boost quality even further. in summary, we contribute a new benchmark for language programs (section 3), the first systematic investigation of the cost-quality tradeoffs of language programs (section 5), and new empirical insights showing that employing appropriate language programs with cheaper models for some tasks is both cost-and performanceoptimal compared to using the raw prediction from a stronger model and offering guidance on which language program architecture and optimizer choices work best (section 6, section 7). ## related work composing language model calls with tool usage into complex applications is now popular, with the help of established language model programming libraries such as dspy @xcite , langchain @xcite , or textgrad @xcite . language programs compose and integrate knowledge and information flow and are often equipped with additional reasoning capabilities. for instance, rag (lewis et al., 2020) combines a language model with external retrieval from a corpus for knowledge-intensive tasks; multi-agent debate @xcite harnesses multiple models debating each other to sharpen mathematical and strategic reasoning; self-refine @xcite iterates on its own outputs for continuous improvement; and react @xcite integrates step-by-step reasoning with actions to facilitate interactions with external environments. recent research also focuses on scaling inferencetime computation by generating multiple responses and verifying them with specialized models @xcite . ## the langprobe benchmark to offer an overview of langprobe, we define different categories of benchmarks and then describe the selected programs and optimizers that we will study in this work. a detailed list of descriptions can be found in appendix b and appendix c. ## benchmarking language programs evaluating language programs with langprobe are simply cartesian products of dataset tasks, language programs, optimizers, and language models. while some language programs obviously do not work with specific types of datasets, we evaluate all such possible combinations. to this end, we evaluate 16 distinct tasks from various open-source datasets with six different language models (gpt-4o, gpt-4o-mini, o1-mini, llama3.1-8b-instruct, llama3.2-3b-instruct, and llama3.3-70b-instruct) and a total of more than 10 different language model programs. we also apply four prompt optimization techniques described in section 3.3 to all language model programs with different hyperparameters ( ## costs of programs and optimizers in this section, we analyze the cost-performance trade-offs of using language models with varying inference costs and capabilities when combined with language programs and optimizers. figure 2 visualizes these trade-offs as pareto curves aggregated over multiple datasets in langprobe. the x-axis uses a logarithmic scale for inference costs to accommodate the wide range of costs across different configurations. we make the following observations from the pareto curves: 1) every point on the paretooptimal curve is instantiable: the pareto-curves are calculated as an upper-left convex hull over linearly scaled performance-cost axes, and hence, every point on or below the curves is instantiable by cost-aware load balancing between the configurations at either end of a pareto segment. 2) use of language programs or optimizers enables achieving better performance at lower cost: the model+program+optimizer pareto curve dominates (achieve better performance at lower cost) the model+program and model+optimizer pareto curves, which in turn dominate the model pareto curve, implying that using language programs and optimizing them has the capacity to support significant improvements not only with respect to performance, but also cost. 3) smaller models with effective programs and optimizers outperform larger models, at a lower cost: we observe that often a smaller or cheaper lm with language program or optimization (or both) outperform configurations with larger or expensive lms lacking language programs or optimization both in terms figure 3 : performance comparison between best-performing programs and the baseline. in most cases, the baseline is a zero-shot call to the lm with task description and task inputs. unoptimized programs are original, unchanged language programs, while optimized language programs are trained with updated prompts to adapt to the specific tasks. scores are averaged from all language models we evaluated, including both openai models and llama models. we also include the same comparison of gpt-4o-mini only in appendix f to reduce possible scattering of averaging results from different model families. figure 8 delivers a similar conclusion as this plot: in almost all tasks, both optimized and unoptimized programs perform better than the raw model prediction baselines. of cost and absolute performance. on aggregate, in figure 2 , we observe that gpt-4o-mini with language programs and optimization achieves 11.68% higher score than gpt-4o's baseline at just 50% of the cost, and gets slightly better performance than gpt-4o with language programs at 10% of the cost. among individual datasets, in figure 9c , we see that gpt-4o-mini with program composition and optimization achieves 33.2% better result than gpt-4o without program composition and optimization at 18% lesser cost. these instances reflect the importance of composing compound systems to prompt language models and optimizing these systems, which can lead to increased downstream task performance at fractional costs. in some instances, we observed that using an optimizer can actually reduce the inference cost for the same program while boosting performance. for instance, in gsm8k (figure 11b ), we see that gpt-4o-mini with optimization (bootstrapfew-shotwithrandomsearch) achieves 8% better result at 5% lesser inference cost than gpt-4o-mini without optimization. since optimizing the program with bootstrapfewshotwithrandomsearch includes few-shot examples in the prompt, which increases the number of input-context tokens presented to the model but excludes the generation of lengthy reasoning traces in the output, and the notion that lm input token costs are cheaper compared to output token costs, we observe such cases. in conclusion, our results demonstrate that language programs and optimizers can significantly enhance performance and cost-efficiency across tasks compared to raw model predictions on the aggregate. ## language program design we now address the research question of what language program architecture achieves higher quality, e.g. whether structured language programs consistently outperform direct model prediction baselines and which programs contribute the most to performance gains. additionally, we examine whether the effectiveness of different language programs varies across tasks, highlighting the conditions under which they provide the greatest benefit. first, we show an overview of performance comparison between best language programs and raw model prediction baseline in figure 3 . in almost all cases, using some selected language programs, either unoptimized or optimized, gives a better performance than the raw model prediction baseline. we present the overall result of language program performance in figure 3 . in this figure, we observe constant improvement compared to the raw model prediction baseline. on some knowledge-intensive benchmarks like hotpotqa or irera, the improvement is significant, with relative gains of +258% and +829%, respectively. in these benchmarks, the raw model lacks certain knowledge to directly answer complex questions or classify complex labels. however, with the addition of using a retriever as a tool, the structured language program is able to fetch additional information from the retriever and integrate the information to the context. this capability allows the program to address knowledge gaps that a raw model would struggle with, enabling accurate and context-aware responses. 6.2 which language program designs provide more improvements? although figure 3 provides a clear overview of the best-performing language programs on each dataset, it does not easily show the relative improvements achieved by different programs. to address this, we present figure 4 , which highlights, for each dataset in the knowledge, reasoning, and math categories, both the best-performing and worstperforming programs and their performance relative to the baseline. from figure 4 , the performance of different language programs varies significantly when compared to the baseline. on the left side, we observe significant improvement over the baseline by mostly retriever-augmented generation programs. conversely, other programs, including generator-criticranker for ragqaarena and generatorcrit-icfuser for judgebench perform worse than the baseline. inspecting the evaluation traces, we find that when the program involves multiple modules, errors (like wrong format, factual errors, or hallucination) cascade from one module flow into other modules, causing errors to aggregate. luckily, the error aggregation patterns, especially parsing errors, are mitigated through carefully curated fewshot examples and instructions from different optimization techniques as highlighted by the positive performance with both optimized variances. m ip r o v 2 b o o t s t r a p f e w s h o t r a n d o m s e a r c h -t r u le in f e r -t m ip r o v 2 -t m ip r o v 2 -li t e -t r u le in f e r m ip r o v 2 -li t e b o o t s t r a p f e w s h o t b o o t s t r a p f e w s h o t r a n d o m s e a r c h r u le in f e r -li t e optimizer 20 0 20 40 60 80 100 120 relative gain% figure 6 : for all the experiments with llama models, the relative gains by the optimizer compared to the same unoptimized program. we report 90th percentile, median, and 10th percentile. optimizers with "lite" suffixes are configured with less compute resources, and optimizers with "t" are configured with a stronger optimizer model (gpt-4o-mini). from this plot, in best case scenarios, all optimizers provide large performance gains of more than 80%. the median performance gains are from 6.3% to 18.1%. however, in some cases, optimizers do degrade the performance of the unoptimized program. overall, these results demonstrate various gains due to compositional program architectures, but they also show that human judgment about which compositions to pursue (or a well-scoped search for tasks with enough data) is still required for best performance. in other words, there is no general "set it and forget it" strategy for program composition within the scope we considered. ## optimizers as in section 6, programs are important for most datasets, as they provide additional context or more opportunities to reason with the language model. in addition to program design and architecture, prompt-based optimizations are also crucial to the performance of a compound ai system. in this section, we discuss whether the existing optimization techniques are effective by answering the following research questions. ## conclusion we presented langprobe, a benchmark for evaluating language programs across a range of tasks. we used it to investigate far more comprehensively than prior work the interactions between architectures, optimization strategies, language models, and the resulting quality and cost tradeoffs. we hope that this starts a line of work on studying this new category of ai systems and that our current findings can already begin to offer practical guidance for researchers and practitioners designing and optimizing modular ai systems. lastly, one of the primary goals of langprobe is to facilitate the development and comparison of new language program architectures and optimization strategies.