Humanity’s Last Code Exam: Can Advanced LLM s Conquer Human’s Hardest Code Competition? (2025)

Abstract: Code generation is a core capability of large language models (LLMs), yet mainstream benchmarks (e.g., APPs and LiveCodeBench) contain questions with medium-level difficulty and pose no challenge to advanced LLMs. To better reflected the advanced reasoning and code generation ability, We introduce Humanity’s Last Code Exam (HLCE), comprising 235 most challenging problems from the International Collegiate Programming Contest (ICPC World Finals) and the International Olympiad in Informatics (IOI) spanning 2010 – 2024. As part of HLCE, we design a harmonized online–offline sandbox that guarantees fully reproducible evaluation. Through our comprehensive evaluation, we observe that even the strongest reasoning LLMs: o4-mini(high) and Gemini-2.5 Pro, achieve pass@1 rates of only 15.9% and 11.4%, respectively. Meanwhile, we propose a novel “self-recognition” task to measure LLMs’ awareness of their own capabilities. Results indicate that LLMs’ self-recognition abilities are not proportionally correlated with their code generation performance. Finally, our empirical validation of test-time scaling laws reveals that current advanced LLMs have substantial room for improvement on complex programming tasks. We expect HLCE to become a milestone challenge for code generation and to catalyze advances in high-performance reasoning and human–AI collaborative programming. Our code and dataset are also public available¹.https://github.com/Humanity-s-Last-Code-Exam/HLCE

## introduction large language models (llms) @xcite @xcite have demonstrated human-level proficiency across a wide range of text understanding @xcite , reasoning @xcite , and generation tasks. among these, code understanding and generation @xcite @xcite have emerged as key research areas, since code inherently reflects reasoning and logical thinking skills. to evaluate llms' capabilities in this domain, many code generation benchmarks, such as live-codebench @xcite , mbpp @xcite , humaneval @xcite , and swebench @xcite , have been established. these benchmarks have accelerated the progress of llm-based code models, but the most advanced llms are now achieving near-perfect or saturated performance on many of these tasks. this raises the question: do current benchmarks truly reflect the advanced reasoning and code generation abilities of state-of-the-art llms, especially when it comes to complex, structured, or algorithmic coding challenges? current benchmarks suffer from several critical limitations: (1) limited difficulty: with the increasing capabilities of llms, many benchmarks have become too easy, as shown in figure 1 . for example, benchmarks like humaneval already report high pass rates, and achieving perfect scores is now a matter of incremental progress. (2) absence of interactive-based evaluation: most contemporary code competition benchmarks (e.g., live-codebench) rely on standard i/o judging, where code submissions are evaluated via input/output pairs. this facilitates outcome-reward reinforcement learning @xcite @xcite , allowing models to exploit these feedback loops. however, interactive-based judging 2 deepseek-r1 openai-o3-mini(high) openai-o4-mini(high) gemini-2.5-pro insufficiently characterized. to address these limitations, we introduce a new benchmark, humanity's last code exam (hlce), comprising carefully curated and rigorously filtered programming problems sourced from the ioi and icpc world finals competitions from 2010 to 2024. after extensive manual cleaning and validation, the benchmark contains 235 high-quality competitive programming problems, each accompanied by comprehensive test cases. hlce extends beyond traditional acm input/output formats to include interactive challenges that demand dynamic program behavior and real-time interaction. using hlce, we evaluate 12 leading llms, including both reasoning and non-reasoning models. our results indicate that even state-of-the-art models o4-mini(high) and gemini-2.5 pro achieve only 15.1% and 11.4% pass@1 rates, respectively, on hlce. this demonstrates that hlce presents a significantly higher level of difficulty compared to existing code generation benchmarks. furthermore, we introduce a novel self-recognition task, in which llms must determine whether their generated code solutions are correct or incorrect, pro-viding direct insight into the models' capacity for self-assessment and error recognition. additionally, hlce enables empirical analysis of test-time scaling laws. our findings reveal that, despite the impressive capabilities of models like o4-mini(high) and gemini-2.5 pro, the upper bounds of test-time performance remain largely unexplored, suggesting considerable room for llm-based reasoning. finally, we consider an intriguing question: can advanced llms genuinely compete with top human programmers in ioi and icpc world finals? to explore this, we collected performance data for medalists from these competitions spanning 2010 to 2024 and directly compared their results to those of leading llms. our analysis shows that models such as o3-mini, gemini-2.5-pro, and o4-mini are capable of achieving medal-level performance in icpc competitions. however, these models still underperform compared to human medalists in the ioi, highlighting the persisting challenges in truly top-tier human-competitive code generation. in summary, our contributions are as follows: ## related work large language models for code generation. recent advancements in llms have significantly enhanced the capability of automated code generation. models such as codex @xcite , starcoder @xcite , and codellama (roziere et al., 2023) have demonstrated remarkable proficiency in understanding and generating programming code across various languages. the emergence of instruction-tuned models like chat-gpt (achiam et al., 2023) and claude (anthropic, 2025) has further pushed the boundaries of code generation capabilities, allowing for more contextually appropriate and functionally correct code outputs. more recently, reasoning-enhanced models have made substantial progress in the code generation domain, with claude-3.7 (anthropic, 2025), deepseek-r1 (guo et al., 2025), and o4-mini (ope-nai, 2025) exhibiting extraordinary capabilities in producing complex, functional code. code generation benchmarks. llm code generation evaluation has evolved through diverse benchmarks. humaneval @xcite established a standard with python function completion tasks, while mbpp @xcite expanded this approach with varied difficulty problems. specialized benchmarks emerged targeting different programming aspects: codecontests @xcite and apps @xcite focus on competitive programming challenges, while ds-1000 @xcite addresses data science tasks. recent developments include live-codebench @xcite for measuring performance on coding competitions across different time periods, aider @xcite ) evaluate llm's ability to follow instructions and edit code successfully without human intervention, and swe-bench @xcite for real-world github issues. open-r1 @xcite maintained a leaderboard of ioi 2024, but only a few problems were prone to causing errors and overfitting. as illustrated in figure 1 , hlce distinguishes itself through its exceptional difficulty level: even the most advanced reasoning-focused llms struggle to perform well, thereby establishing a new ceiling for evaluating the code generation capabilities of current models. ## hlce benchmark as illustrated in figure 2 , the hlce framework comprises three principal components: dataset, evaluation task, and evaluation framework. ## experiments in this section, we evaluate and analyze the performance of current state-of-the-art llms on two tasks from the hlce benchmark. ## discussion the future direction of code llm capabilities the code llm landscape is rapidly evolving with divergent approaches to balancing general coding capabilities and competitive programming skills. claude 3.7 appears to prioritize general code generation over specialized algorithmic performance, possibly reflecting anthropic's assessment of market demand for practical development assistance. conversely, gemini 2.5 pro excels in both domains, suggesting that with sufficient scale and sophistication, the trade-off between these capabilities may be less constraining than previously thought. therefore, general code abilities and competitive programming abilities in future llms are not mutually exclusive directions. llm performance in programming competitions remains an important metric for gauging current progress toward agi. the future direction of test-time scaling in code llm with the powerful performance demonstrated by gemini 2.5-pro and o4-mini (high), concerns have emerged about whether testtime scaling laws, similar to pre-training scaling laws @xcite , are approaching their limits. our experiments demonstrate that testtime scaling substantially impacts model performance. while these results are impressive, we have likely not reached the ceiling of potential improvements through such techniques. as models continue to scale in test time, we anticipate further advancements in reasoning efficiency @xcite and efficacy, and generating sophisticated algorithmic solutions. ## conclusion in this work, we introduced hlce, a challenging benchmark of 235 competitive programming problems from ioi and icpc world finals. top models achieve only 15.1% and 11.4% pass@1 rates. our benchmark includes standard and interactive programming challenges alongside a novel selfassessment task. by incorporating human competition data, we established metrics comparing llms with top-tier programmers, revealing substantial room for improvement. test-time scaling law validation confirms current models have not reached performance ceilings, suggesting promising directions for advancing llms' reasoning in complex programming tasks. hlce aims to drive progress toward code llms that reach the proficiency level of elite human competitors.