Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies (2024)

Abstract: A diverse array of reasoning strategies has been proposed to elicit the capabilities of large language models. However, in this paper, we point out that traditional evaluations which focus solely on performance metrics miss a key factor: the increased effectiveness due to additional compute. By overlooking this aspect, a skewed view of strategy efficiency is often presented. This paper introduces a framework that incorporates the compute budget into the evaluation, providing a more informative comparison that takes into account both performance metrics and computational cost. In this budget-aware perspective, we find that complex reasoning strategies often don’t surpass simpler baselines purely due to algorithmic ingenuity, but rather due to the larger computational resources allocated. When we provide a simple baseline like chain-of-thought self-consistency with comparable compute resources, it frequently outperforms reasoning strategies proposed in the literature. In this scale-aware perspective, we find that unlike self-consistency, certain strategies such as multi-agent debate or Reflexion can become worse if more compute budget is utilized.

## introduction the arena of large language models (llms) such as @xcite @xcite has seen a proliferation of diverse reasoning strategies. however, comparing these strategies fairly and comprehensively has proven to be a challenging task due to their varied computational requirements. for instance, strategies like tree of thoughts (tot) necessitate branching out into multiple sequences and incorporating self-evaluation, making them more * work conducted during an internship at amazon † work conducted while at amazon compute-intensive than others. therefore, an evaluation framework that only accounts for performance metrics may miss crucial practical factors such as computational cost. in this paper, we propose the inclusion of the compute budget into the performance measurement of different reasoning strategies. this budget-aware comparison yields a more balanced perspective on the effectiveness of reasoning strategies, accounting for both the quality of the output and the computational resources expended. our empirical research uncovers a significant correlation between the performance and the compute budget. we find that a straightforward baseline strategy, chain-of-thought reasoning coupled with self-consistency, can be remarkably competitive. when scaled to match the compute resources of more sophisticated methods such as multi-agent debate (mad) @xcite , reflexion @xcite , plan and solve @xcite , least to most prompting @xcite , progressive hint prompting @xcite , this baseline strategy often outperforms them in achieving the best trade-off between performance and budget. we further investigate the reasons behind the gap from simple cot sc and other reasoning strategies by providing both empirical and theoretical evidence. then we scrutinize the influence of two specific types of budgets on performance: (1) the answer generation budget, and (2) the evaluation budget. the success of reasoning strategies that leverage self-evaluation is model/dataset-dependent and strongly correlated with calibration. this work provides a robust framework for comparing a wide array of reasoning strategies and illuminates the significance of self-evaluation in these models. we hope this sets the stage for more focused research on efficient budget utilization and paves the way for the development of even more effective reasoning strategies. (2) the scale-aware comparison between reflexion and sc also illustrates the artifact of scale on performance. for both datasets, we show both budgets, the number of total tokens, and the number of queries. all results were obtained from gpt-3.5. concretely, our contributions are • we introduce a budget-aware evaluation framework spanning three dimensions: queries, tokens, and monetary cost, advocating for the token-based metric as the most holistic. this metric adeptly captures both the latency and financial implications of computational tasks. ## inference budget of reasoning strategies while the raw performance of different prompting or reasoning strategies for llms is a common topic, how different strategies perform when budget-aware is less well-studied. however, taking budget into account can be critical when using llms. in this section, we describe different usage scenarios that a user could be interested in and what budgetary metrics would be relevant to those scenarios. ## a critical evaluation in budget-aware environments this section explores key components that can make reasoning strategies successful from the budget-aware perspective. first, we show that the inference budget is often overlooked but is one of the primary indicators of the success of a reasoning strategy. we show that from the budget-aware evaluation perspective, cot (or variants of it like plan and solve, least to most) self-consistency, for instance, is a strong baseline that can outperform or match many proposed reasoning strategies in the literature given the same level of budget. experiement setup we use existing reasoning strategies in literature to perform this study, namely multi-agent debate (mad) @xcite , reflexion @xcite , plan and solve @xcite , least to most prompting @xcite , progressive hint prompting @xcite , and tree-of-thoughts @xcite . we conducted our experiments across a diverse range of reasoning tasks, utilizing math reasoning datasets such as gsm8k @xcite , math @xcite , and theo-remqa @xcite , along with the commonsense reasoning task csqa @xcite , and multi-hop reasoning task hotpotqa @xcite (see appendix a.1). additionally, we performed an in-depth analysis of the puzzle game game24 @xcite . for models, we use mistral-7b-instruct, llama-2-70b-chat, mixtral-8x7b-instruct, gpt-3.5, and gpt-4. (see appendix a.1 for more details about model hyperparameters) metrics to provide a fair comparison, we control for the cost/budget of each reasoning strategy to have the same level of budget. specifically, we demonstrate two budgetary metrics which built on what was discussed in section 3.1: each reasoning strategy may have different ways of increasing the budget depending on its design. for cot sc, we increase the number of sampled cot paths to increase queries/tokens. for mad, we set the number of agents to six and increased the number of rounds up to three, maxing at 18 queries. for reflxion, we increase the number of proposals of answers and reflections to increase the budget. ## what makes reasoning strategies work we provide a detailed analysis of reasoning strategies. specifically, we examine the causes of the performance gaps identified in our budget-aware evaluations among strategies such as mad, reflexion, and self-consistency in section 5.1. in section 5.2, we analyze the components of tree-ofthoughts. finally section 5.4 assesses the role of self-evaluation in the reasoning loop. ## conclusion in this paper, we examined the performance of seven reasoning strategies on the often overlooked metric of budget. we used budget metrics of queries and tokens to reflect various ways llms are used (llm apis or self-host). we identified self-evaluation as an important aspect of many reasoning strategies and analyzed different prompting strategies to have the model evaluate its generations. we then evaluated self-evaluation and found that although self-evaluation could be promising at improving performance while being cost-effective, current llms are mostly incapable of doing that. with the current popularity of reasoning strategies, we think this more balanced budget-aware metric is beneficial for the community and helps set the correct trajectory for future llm research.