C o C o ST : Automatic Complex Code Generation with Online Searching and Correctness Testing (2024)

Abstract: Large Language Models have revolutionized code generation ability by converting natural language descriptions into executable code. However, generating complex code within real-world scenarios remains challenging due to intricate structures, subtle bugs, understanding of advanced data types, and lack of supplementary contents. To address these challenges, we introduce the CoCoST framework, which enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement. Moreover, CoCoST serializes the complex inputs and outputs to improve comprehension and generates test cases to ensure the adaptability for real-world applications. CoCoST is validated through rigorous experiments on the DS-1000 and ClassEval datasets. Experimental results show that CoCoST substantially improves the quality of complex code generation, highlighting its potential to enhance the practicality of LLMs in generating complex code.

## introduction automatic code generation from natural language descriptions is becoming more realistic, as large language models (llms) show their potential to generate accurate code @xcite @xcite . various methods have been proposed to improve the quality of llm code generation, such as retrieving offline documents @xcite and debugging generated code @xcite . however, complex code generation is a more difficult task, which involves intricate problem description, sophisticated code logic, and advanced data types @xcite the contributions by xinyi he, jiaru zou and yun lin have been conducted and completed during their internships at microsoft. † corresponding author. 2023; @xcite . the existing methods struggle to address the arising challenges: challenge 1: offline documents cannot meet the diverse demands of code generation. in real-world scenarios, these demands often exceed the capabilities of limited offline documents. for example, problem descriptions may involve functions that are not covered by pre-collected documents. additionally, complex code generation for diverse needs often entails highly complex logic and a series of transformation functions like the programming problem in figure 1 , where simple api examples in documents fail to provide adequate guidance. challenge 2: in real-world situations, there is often a shortage of test cases (e.g., test cases in figure 1 ) for automatic code generation. most existing work depends heavily on pre-existing test cases in datasets @xcite , which are difficult to acquire directly in practical scenarios. challenge 3: hidden bugs in complex code require meticulous identification and refinement. current techniques frequently enhance code by analyzing execution errors @xcite . but in the case of complex code, the executable code sometimes contains hidden bugs like the highlighted part of the initial code in figure 1 . to address these challenges, we introduce a new code generation framework named cocost foot_0 (automatic complex code generation with online searching and correctness testing) that improves the generation and refinement of complex code by llms through the planned online searching and automatic correctness testing steps. the intuition of cocost is straightforward: during the coding process, most human developers are not bothered by the above challenges, as illustrated in figure 1 . developers can easily overcome these obstacles by i am building a custom metric to measure the accuracy of one class in my multi-class dataset during training. i am having trouble selecting the class. the targets are one hot. i have 10 classes in total, so i need a n*10 tensor as a result. now i have a list of integers, how to get a tensor like: searching online through engines (e.g., google and bing) for solutions, experiences, and guidelines. in addition, they can create test cases and execute code to ensure the correctness of the code logic. to address challenge 1, cocost proposes an online search methodology. this process involves querying web search engines and then extracting pertinent information to construct llm prompts. the approach presents several benefits: (1) retrieving information from the up-to-date blogs or q&a platforms, such as stackoverflow, facilitates the emulation of commonly used code patterns, thereby reducing the complexity of generated code. (2) online search extends beyond the scope of static offline documentation, covering a wider range of problems without being confined to a predetermined set. meanwhile, it reduces the effort developers need to expend in assembling documentation, thereby increasing the framework's level of automation. using problem descriptions as search queries can be difficult, because problems are generally intricate and include several components. therefore, we propose an online search with query generation through planning. to address challenge 2, we introduce generation of test cases during refinement. several studies @xcite have attempted to generate tests. however, these methods often fall short when applied to the generation of complex code due to its intricate logic and outputs, which complicate the direct production of accurate tests (both the inputs and expected outputs for the solution code). cocost utilizes llms to automatically generate test cases (the inputs for the code). this strategy cleverly focuses on generating test cases without attempting to produce complete tests. it significantly simplifies the process of test case generation and facilitates its precise creation for complex code. to address challenge 3, this work prioritizes correctness testing in refinement. during the refinement process, it is more critical to verify that the executed code produces the correct results rather than just checking the existence of the errors. co-cost incorporates both the execution output results and the errors within the refinement prompts for llms to enhance the correctness. moreover, during refinement, sophisticated data types and structures (within complex code itself, its inputs, and its execution results) are challenging for llms to understand, e.g., large pandas dataframes, and matplotlib charts. thus, cocost proposes serialization of input and output to convert them into understandable sequences before being processed by llms. particularly those are excessively long or non-textual modalities. we evaluated the effectiveness of cocost on two complex code generation datasets (ds-1000 and classeval). compared with the existing stateof-the-art (sota) baseline, we achieve a 7.8% improvement on ds-1000 and an average of 9.47% on classeval.moreover, we analyze and discover that cocost requires models to have different capabilities such as planning, which vary according to the complexity of the problem. in summary, our main contributions are as follows. code generation datasets. the realm of automated code generation has been propelled by benchmark datasets such as humaneval @xcite , mbpp @xcite , and apps @xcite , which assess the proficiency of language models in generating executable code from descriptions. these datasets encompass a variety of programming problems, yet recent studies have sought to escalate the complexity of code generation tasks. works like ds-1000 @xcite , classeval @xcite and text2analysis @xcite have introduced datasets targeting specialized domains, including data science, object-oriented class generation, and data analysis. these endeavors reflect an emerging trend towards enhancing models' abilities to produce sophisticated and domain-specific code structures. in this paper, we select datasets with complex code generation to evaluate cocost. ## methodology the code generation task involves predicting a solution code w given a problem description d. when given an input i, the execution of code w produces an output result o and a potential error e, where both o and e can be empty ∅. the generated codes are evaluated against a set of test cases and ground truth {(t j , g j )} j @xmath0. the correctness of the code w is determined by verifying o @xmath1, @xmath2, . . . , j}. in this work, we adopt a two-step approach for code generation, mirroring the way humans write code. the first step is retrieval, where relevant information is obtained through an online search and utilized by llms to generate initial code. the second step is refinement, where the initial code is refined based on the execution results, leading to the generation of the final version of the code. llm plans 1. iterate over each column in the dataframe. 2. for each column, calculate the value counts. 3. … plans & queries online search step 1: retrieve with online searching queries 1. no need to search. 2. need to search, <@xmath3. … .. … ndarray.max(@xmath4, @xmath5, @xmath6, @xmath7, @xmath8.dataframe.apply#... dataframe.apply( _func_ , _axis … … document info stackoverflow title: … stackoverflow question:… stackoverflow top answers:… … qa info query n… query 2… query 1… … step 2: refine with correctness testing situation 1: output & serialize situation2: error & online search numpy array: (16 x 5), … min: 0.00, max: 1.00. serialized content numpy array: (16 x 5), … min: 0.00, max: 1.00. serialized content array([[0. 1. 0. 0. 0.] …]) array([[0. 1. 0. 0. 0.] …]) ## conclusion in this paper, we propose cocost, a novel framework for generating complex code in real-world scenarios by emulating human coding processes like online searching and test case creation. it effectively overcomes challenges in code structure and logic, subtle bug detection, and handling of complex data. the framework's innovative use of online search, planning for query generation, correctness testing, and input-output serialization significantly improves code accuracy and model understanding. tested on various datasets, cocost outperforms existing methods, demonstrating its efficacy in real-world code generation tasks.