O pen C ode I nterpreter: Integrating Code Generation with Execution and Refinement (2024)

Abstract: The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4’s 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreterbrings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.

## introduction code generation has been a pivotal challenge within computer science for several decades. recently, the landscape of code generation has been revolutionized by the advent of large language models (llms) pre-trained on extensive code corpora @xcite @xcite @xcite @xcite . these models have showcased remarkable capabilities in generating code that accurately aligns with * equal contributions. ## code-feedback in this section, we detail the creation of our code instruction tuning dataset, code-feedback (figure 2 ), designed to train opencodeinterpreter. code-feedback is crafted to meet specific criteria: 1) diverse and challenging real-world queries: the dataset should encompass a wide range of queries derived from real-world coding tasks, presenting both diversity and complexity. 2) multi-turn dialogue structure: code-feedback is structured as multi-turn dialogues, incorporating two types of feedback: execution feedback, which includes outputs and diagnostics from compilers, and human feedback, consisting of additional guidance or instructions from users. 3) interleaved text and code responses: each response is expected to provide responses that blend natural language explanations with code snippets, offering a holistic approach to solving coding queries. to assemble a dataset that fulfills these desiderata, we have employed five distinct methods. examples of these five categories can be found in appendix e. the sources of our queries fall into two main categories: a variety of open-source datasets and coding challenges from leetcode. in the next subsections, we will discuss how we develop data construction methods to meet the three aforementioned criteria from the two data sources. ## main results this section reports opencodeinterpreter and baselines in single-turn and multi-turn code generation settings. the results are in our experimental analysis reveals opencodeinterpreter's strong performance, with several configurations matching or surpassing leading benchmarks. the opencodeinterpreter-ds 33b variant achieves the highest scores among open-source models. this accomplishment is remarkable, especially considering the significant presence of lowquality or incorrect data in the initial training set. ## conclusion in conclusion, opencodeinterpreter represents a significant leap forward in the field of code generation, bridging the previously identified gap between open-source models and the advanced capabilities of proprietary systems like the gpt-4 code interpreter. by integrating compiler diagnostics and human feedback into an iterative refinement process, opencodeinterpreter not only surpasses traditional one-off generation approaches but also introduces a level of adaptability and precision previously unseen in open-source models. the introduction of code-feedback, with its extensive multi-turn interactions, further empowers opencodeinterpreter to dynamically refine code in response to evolving user intents and complex coding tasks.