Competition-Level Problems are Effective LLM Evaluators (2024)

Abstract: Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently. This paper aims to evaluate the reasoning capacities of LLMs, specifically in solving recent competition-level programming problems in Codeforces, which are expert-crafted and unique, requiring deep understanding and robust reasoning skills. We first provide a comprehensive evaluation of GPT-4’s perceived zero-shot performance on this task, considering various aspects such as problems’ release time, difficulties, and types of errors encountered. Surprisingly, the perceived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems, which shows the potential data contamination, as well as the challenges for any existing LLM to solve unseen complex reasoning problems. We further explore various approaches such as fine-tuning, Chain-of-Thought prompting and problem description simplification. Unfortunately, none of them is able to consistently mitigate the challenges. Through our work, we emphasize the importance of this excellent data source for assessing the genuine reasoning capabilities of LLMs, and foster the development of LLMs with stronger reasoning abilities and better generalization in the future.

## introduction the rise of llms has generated significant interest in the artificial intelligence community. these models, notably @xcite , have displayed impressive reasoning capabilities that are being harnessed in various fields @xcite . however, questions 1 have been raised about how to accurately evaluate the reasoning abilities of llms and the extent of data contamination issues @xcite . regarding these issues, our study aims to assess the reasoning capabilities of llms through their ability to generate algorithms for solving competition-level programming problems. these questions are meticulously crafted by experts to form rigorous competitions. they possess high quality, are unique, and exhibit excellent discriminative ability. the testing cases are also meticulously prepared. this necessitates that llms deduce the solution from the presented scenario, which requires a thorough understanding of algorithms, combined reasoning and coding skills, and strong problem-solving abilities. these problems thus present a significant challenge to both human coders and llms. consequently, competition-level programming problems serve as effective tools for evaluating the two issues previously discussed: they assess the reasoning abilities of llms and, due to the strict problem selection process in competitions, reduce the likelihood of data contamina-tion in new problems. our research provides an in-depth analysis of the zero-shot performances of gpt-4 and other code llms on competition-level programming problems in codeforces, considering factors such as release time, problem difficulty, and the types of errors encountered. the main insights of our study include: (1) gpt-4 performs significantly worse on programming problems released after september 2021, casting doubt on its actual reasoning abilities. (2) gpt-4 shows limited capability to solve difficult problems, indicating potential weaknesses in complex problem-solving. (3) gpt-4 struggles with the first test case, suggesting errors may stem from its understanding of the problem at hand. (4) the related phenomenon can be also observed in other llms, indicating that insufficient reasoning ability may be a common problem. to explore possible ways to enhance the zeroshot performances of these llms on competitionlevel programming problems, we investigate several methods to improve performance on unseen problems. these methods include supervised fine-tuning with code-specific llms, chain-of-thought prompting @xcite , and problem statement simplification. specifically, we fine-tuned codellama @xcite and @xcite , which are specialized language models designed to handle programmingrelated tasks. however, none of these methods consistently mitigated the issue or resulted in noticeable performance improvements, particularly for more difficult problems. this finding indicates that difficult and unseen programming problems are effective evaluators of llms. overall, the primary contributions of this study lie in proposing and validating that recent competition-level programming problems serve as an excellent data source for assessing the genuine reasoning capabilities of llms. we aim to foster further research in this field by innovating new approaches to address the challenge of complex reasoning problems in llms and by establishing reliable evaluation benchmarks for llms that minimize the risk of data contamination. ## one step forward in this section, we explore some approaches to mitigate the poor performance on unseen problems. ## conclusion in this study, we utilize competition-level programming problems from codeforces to analyze the reasoning capabilities of llms. we find a significant decrease in perceived performance of gpt-4 on unseen problems, consistent across a range of difficulties, problem types, and experimental settings. this decrease highlights concerns of data contamination in benchmarks and the need for unseen tasks to properly assess llms' reasoning ability with complex challenges. our research also extends these insights to other open-source llms, revealing the common difficulties these models face with complex, previously unencountered reasoning tasks. this is indicative of the llms' intrinsic limitations in reasoning. as a primary probe, we explore several straightforward strategies, but none of them consistently mitigated the issues. through our work, we hope to emphasize the critical need for robust datasets to accurately evaluate llms' reasoning abilities and to inspire advancements in llms that demonstrate improved reasoning abilities.