Evaluating LLM s Efficiency Using Successive Attempts on Binary-Outcome Tasks (2025)

Abstract: Evaluating Large Language Models (LLMs) using single-attempt metrics like Success Rate (SR) overlooks their capacity for iterative problem solving. In tasks with binary outcomes (success or failure), such as coding or planning, LLMs often benefit from multiple attempts. Existing multiattempt metrics like pass@k and success@k account for eventual success but ignore how efficiently it is achieved, making them more costly. We propose a new evaluation method with Successive Multiple Attempts, where a maximum number of retries is fixed, and introduce our Success Efficiency (SE) metric, which captures both success and efficiency in a single value by rewarding earlier successes and penalizing delays. Tested using the HumanEval dataset across six LLMs, SE captures how quickly an LLM solves tasks, which existing metrics do not offer. This work complements existing evaluation methods by measuring not only whether LLMs succeed but also how efficiently they do so.

## introduction understanding the practical utility of large language models (llms) requires evaluating them across a wide range of tasks. many of these tasks, including code generation @xcite , theorem proving @xcite , logical reasoning (yu et al., 2020), and instruction following @xcite , can be framed as binary outcome tasks, where each evaluation result of an llm is either a success or a failure. traditionally, such tasks have been evaluated using a single-shot paradigm, in which an llm is given a single attempt to solve a task. this yields simple metrics like the success rate (sr) described in section 2, which captures the rate at which the llm succeeds on the first attempt across a given list of tasks. while sr and other similar metrics are informative, they overlook an important dimension of llms behavior : the capacity for multi-attempt problem solving @xcite . in practical deployments, such as interactive assistants @xcite , automated coding tools @xcite , or planning agents @xcite , llms are often allowed multiple chances to revise or regenerate responses. to reflect this, recent studies have introduced evaluation frameworks that support multiple attempts, either through independent sampling, such as pass@k @xcite or sequential retries, such as success@k @xcite . these metrics assess whether an llm can solve a task within a fixed number of attempts, but treat all successful completions equally, regardless of whether the solution occurred early or late in the sequence. consequently, they fail to capture the efficiency of problem solving (as defined in section 2), which is a critical factor in latency-sensitive and resource-constrained applications. to address this limitation, we propose an evaluation methodology based on successive multiple attempts, in which an llm is allowed to iteratively attempt a task up to a fixed maximum number of attempts (maxa). this controlled retry setting enables us to measure not only whether an llm can eventually solve a task, but also how rapidly and it does so. at the core of our methodology lies the success efficiency (se) detailed in section 2, a novel metric that unifies both success and efficiency into a single, interpretable measure. unlike metrics such as pass@k or success@k, se provides a continuous score that penalizes delayed success. a detailed comparison between our proposed se metric and these related metrics is given in section 3. we evaluated six state-of-the-art (sota) llms using our methodology on the humaneval dataset @xcite , a widely used benchmark for code generation. the evaluation setup and results are detailed in section 4, where we compared se to the traditional sr metric. the results show that se provides a more informative assessment by distinguishing between llms that rapidly succeed and those that required multiple retries. this underscores se's value as a metric for evaluating llms in iterative, retry-enabled scenarios. ## evaluation methodology to rigorously evaluate the performance of llms on a fixed set of tasks, we introduce three complementary metrics. these metrics assess both the effectiveness (how often the llm succeeds) and the efficiency (how quickly it succeeds within a limited number of attempts). let @xmath0. the evaluation metrics are defined as follows : the se score combines both success and efficiency into a single value. for any @xmath1, se reaches its maximum (equal to sr) when @xmath2, meaning all successful tasks were completed on the first attempt. in contrast, it reaches its minimum when @xmath3, reflecting highly inefficient completions. the exponent in the formula of se, avgtca-1 maxa-1 , measures how far the llm's avgtca deviates from the ideal value of 1 (perfect efficiency) normalized by the maximum possible deviation (maxa-1), within the sr calculation, this exponent acts as a normalization penalty in [0, 1], applying minimal penalization when (@xmath4, and gradually increasing as avgtca approaches the allowed maximum (maxa). this formulation ensures that se remains : -bounded : it cannot exceed sr or fall below zero, preserving numerical stability. ## existing evaluation metrics review existing evaluation metrics for binary-outcome tasks, such as sr, typically measure whether a task is eventually solved. some metrics, like pass@k and success@k, account for the number of attempts, but they do not capture how efficiently a solution is reached in terms of successive attempts. in our proposed se metric addresses this gap by integrating both success and efficiency into a single metric. as detailed in section 2, it rewards llms that solve tasks quickly and penalizes those that require more attempts, making it especially suited for evaluating llms in practical applications that enables attempting. ## conclusion and future work in this paper, we propose a novel evaluation metric, success efficiency (se), to assess the performance of llms in multi-attempt settings, capturing both task completion and efficiency. this makes se particularly suitable for practical applications where responsiveness and resource usage are critical. experiments on the humaneval dataset demonstrate se's ability to highlight performance differences which are overlooked by existing metrics. future work may includes extending se to non-binary tasks like open-ended generation or multi-step reasoning, developing adaptive retry strategies based on task complexity, and integrating cost-awareness such as inference time or compute usage, to better promote sustainable llm deployment.