C ode C ontests+: High-Quality Test Case Generation for Competitive Programming (2025)

Abstract: Competitive programming, due to its high reasoning difficulty and precise correctness feedback, has become a key task for both training and evaluating the reasoning capabilities of large language models (LLMs). However, while a large amount of public problem data, such as problem statements and solutions, is available, the test cases of these problems are often difficult to obtain. Therefore, test case generation is a necessary task for building large-scale datasets, and the quality of the test cases directly determines the accuracy of the evaluation. In this paper, we introduce an LLM-based agent system that creates high-quality test cases for competitive programming problems. We apply this system to the CodeContests dataset and propose a new version with improved test cases, named CodeContests+. We evaluated the quality of test cases in CodeContests+. First, we used 1.72 million submissions with pass/fail labels to examine the accuracy of these test cases in evaluation. The results indicated that CodeContests+ achieves significantly higher accuracy than CodeContests, particularly with a notably higher True Positive Rate (TPR). Subsequently, our experiments in LLM Reinforcement Learning (RL) further confirmed that improvements in test case quality yield considerable advantages for RL.

## introduction competitive programming is widely recognized as an important benchmark for evaluating the reasoning and coding capabilities of llms @xcite . solving complex competitive programming problems requires strong reasoning capabilities, as well as mastery of a wide range of algorithms, data structures, and mathematical knowledge. more importantly, competitive programming problems are objectively verifiable tasks; thus, they are not only widely used for benchmarks, but they can also provide accurate rewards for reinforcement learning and serve as a vital data foundation for training large reasoning models @xcite . existing open-source competitive programming datasets usually collect problems from competition platforms like codeforces @xcite , leetcode, and atcoder. however, these platforms do not publicly release their test cases. consequently, although large amounts of problems with statements and solutions are publicly available, the lack of test cases prevents these problems from being effectively constructed into rl training datasets. therefore, existing open-source datasets typically have to construct test cases using various methods. common methods include mutation, as well as using llms to output test cases, etc. however, these methods still have shortcomings in the following two aspects: limited coverage. some methods, such as mutation, can only blindly construct large amounts of random data. they struggle to generate tricky cases and corner cases that require deep algorithmic understanding to discover, making it hard to cover deep-level, complex boundary conditions or special situations within the problem's logic. additionally, some methods, like directly outputting test cases via llms, are often unable to generate large-scale test cases. therefore, they cannot identify incorrect solutions that are logically correct but fail due to excessively high time or memory complexity. limited coverage may lead to false positives, meaning that incorrect solutions might be judged as correct. incorrect test cases. as shown in fig. 1 , a typical programming problem usually imposes constraints on the test cases themselves. existing methods struggle to ensure that the generated test cases can satisfy these constraints, and incorrect test cases can simultaneously lead to both false positives and false negatives. to the best of our knowledge, the issue of incorrect test cases has not received attention in previous research. we examined the test cases in the codecontests dataset and found that incorrect test cases are one of the main causes of inaccurate evaluation. in this paper, we propose an llm-based agent system for constructing test cases for programming problems that have more comprehensive coverage and better correctness. this will allow for further improvement in the quality and scale of code rl datasets. specifically, to address the above two limitations, we propose the following solutions: an agent for test case generation. we designed a generator agent that writes a generator program for each problem to specifically construct diverse test cases, including random data, corner cases, and tricky cases, thereby fully testing various possible solutions and potential error patterns, as well as examining the efficiency of the algorithm through large test cases. this generator program can be run any number of times with different random seeds, thus obtaining any number of test cases, further improving coverage. an agent for test case validation. although this generator agent consciously attempts to satisfy the constraints in the problem, it still has a noticeable probability of making mistakes. therefore, we designed a validator agent. this agent writes a validator program to check whether the input of the generated test cases satisfies all the constraints in the problem. incorrect test cases and the specific reasons for the errors will be fed back to the generator for revision until all test cases satisfy the conditions. the contributions of this paper is summarized as follows: ## related work since most competitive programming platforms do not disclose their test cases, constructing test cases is one of the primary bottlenecks in building code datasets. the test case generation methods currently employed by existing code datasets can generally be categorized into three types: manual generation, mutation-based generation, and llmbased generation. manual. representative works that use manually constructed test cases include mbpp @xcite , humaneval @xcite , and livecodebench @xcite . there are slight differences among these three: the test cases in mbpp and humaneval are handcrafted, resulting in a smaller quantity and insufficient coverage. in contrast, for a portion of the problems in live-codebench, test cases are constructed by human experts specifically targeting the problem characteristics, leading to better coverage. @xcite benchmark 164 ☀ ✗ handcrafted usaco @xcite benchmark 307 ☀☀ ✗ publicly accessible livecodebench @xcite benchmark 511 ☀☀ ✗ semi-automatic apps @xcite train 10000 ☀☀☀ ✗ crawled codecontests @xcite train 13610 ☀☀☀ ✗ mutation taco @xcite train 26433 ☀☀☀ ✗ output by llm the common drawbacks of manually constructing test cases are their high cost, lack of automation, and difficulty in scaling up. therefore, such methods are only suitable for building small-scale evaluation sets and are too costly to use for constructing large-scale training sets. mutation-based. @xcite identified the issue of high false positive rates (fpr) in mbpp and humaneval due to their small number of test cases. they proposed "type-aware input mutation" to generate new test cases by recombining a few existing test cases. a similar mutation approach was also employed to construct the codecontests @xcite dataset. the advantage of mutation methods lies in their complete automation, allowing for the generation of a large volume of test cases. however, their limitation is that if a problem involves complex constraints, mutation often fails to satisfy these constraints, thereby introducing incorrect test cases and leading to a high false negative rate (fnr). according to our verification, approximately one-third of the test cases in codecontests are incorrect. ## the generator-validator agent system to simultaneously address the challenges of correctness and coverage in test case construction, we propose a generator-validator agent system. the generator agent is an llm-based agent that writes test input generators based on the problem description. the validator agent is also an llm agent that writes test input validators to supervise the generator agent, ensuring that the test input generated by the generator agent conforms to the problem constraints. this agent system can automatically build a large-scale and rigorous dataset for llm training by leveraging publicly available data such as problem statements and ground truth solutions. an overview of the workflow is presented in fig. 2 . specific implementation details for the generator and validator are presented below. ## codecontests + : a competitive coding dataset with verified test cases codecontests @xcite stands as one of the largest and most widely recognized competitive coding datasets. it collects a large number of problems, authentic contestant submission records, and generates numerous additional test cases through mutation. however, test cases generated through mutation are often of low quality and may yield unreliable evaluation results, such as misclassifying incorrect solutions as correct and vice versa. in this section, we present our methodology for constructing an enhanced dataset, codecontests + , by building upon the original codecontests. section 4.1 outlines the data cleaning procedures we implemented. section 4.2 details how we utilized the g-v agent system to generate higher-quality test cases for codecontests + . section 4.3 describes the development of customized checkers for problems that accept multiple valid solutions. following this, we compare the quality of codecontests + and codecontests across two primary dimensions. first, in section 4.4, we verify the quality of the test cases of both datasets. more specifically, we utilized 1.72 million authentic contestant submissions, which comprise both correct and incorrect ones, to assess the performance of both datasets in discriminating between correct and incorrect solutions. then, in section 4.5, we employed each dataset to train a 32b llm through grpo, to evaluate the impact of dataset quality on rl training efficacy. ## conclusion and future work in this paper, we propose an llm-based generator-validator agent system capable of leveraging public problem data to construct high-quality test cases for competitive programming problems. this system facilitates the scaling up of high-quality code datasets. using this agent system, we have developed codecontests + by enhancing the original codecontests dataset with better test cases. experimental results demonstrate that the test cases in codecontests + are of significantly higher quality than those in codecontests, and that codecontests + also exhibits substantial advantages in rl training. according to our rough estimate, there are more than 100,000 coding problems with publicly available problem statements and ground truth solutions. therefore, our proposed agent system will enable the full utilization of these data resources, thereby laying the data foundation for further enhancing the reasoning and coding capabilities of llms.