H uman E val Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025)

Abstract: In this paper, we present HumanEval Pro and MBPP Pro, a series of benchmarks to evaluate LLMs on self-invoking code generation task. This task involves providing LLMs with a base problem alongside a related, more complex problem. The models must solve the base problem and leverage its solution to address the more complex one, thereby showcasing their capacity for progressive reasoning and problem-solving. This work features three key contributions. First, we propose a general recipe for generating more challenging versions of existing benchmarks. Second, from the analysis of experimental results over twenty large language models (LLM) on our benchmarks, we have two important observations: (i) Most LLMs excel in traditional code generation benchmarks like HumanEval and MBPP, but their performance declines on self-invoking tasks. For example, o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro. (ii) On self-invoking code generation task, the instruction-tuned models demonstrate only marginal improvements compared to the base models. Third, we disclose the types of failure modes that exist in our evaluation results. All these results underscore the need for further advancements in this area and provide a new prospective to future research.

## introduction large language models (llms) have demonstrated significant progress in various code-related tasks including code generation @xcite @xcite , program repair @xcite , and code translation @xcite , etc. traditional human-annotated benchmarks such as humaneval * ## benchmark construction to facilitate a meaningful comparison between self-invoking code generation and traditional code generation, we have crafted two new benchmarks, humaneval pro and mbpp pro. these benchmarks are extensions of the original humaneval and mbpp, requiring the model to solve both the base problem and a more complex self-invoking problem. in addressing the self-invoking problems, llms are required to apply the solutions they have independently generated for the base problem. this evaluation of self-invoking code generation offers deeper insights into the programming capabilities of llms, extending beyond the scope of singleproblem code generation. the benchmark construction process, illustrated in figure 2 , will be discussed in detail in the following subsections. ## experiment and analysis following previous work @xcite , we use the pass@k @xcite score as the evaluation metric of humaneval pro and mbpp pro. we use greedy decoding strategy to generate solutions for all open-source models and set temper-@xmath0.2 for all api-models (model information is shown in appendix e). for all previous benchmarks, we use the reported results whenever available; otherwise, we evaluate using the evalplus codebase @xcite . ## conclusion we present humaneval pro and mbpp pro, a series of benchmarks to evaluate llms on self-invoking code generation task where the llms are employed to solve the base problem and use its solution to address more complex problems. our evaluation of over 20 llms reveals that, despite notable advancements in traditional code generation, these models still face challenges in self-invoking code generation. humaneval pro and mbpp pro are positioned to serve as valuable benchmarks for code-related evaluations and to inspire future llm development by shedding light on current model shortcomings and encouraging innovation in training recipe.