AMR -Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024)

Abstract: The impressive performance of proprietary LLMs like GPT4 in code generation has led to a trend to replicate these capabilities in open-source models through knowledge distillation (e.g. Code Evol-Instruct). However, these efforts often neglect the crucial aspect of response quality, relying heavily on teacher models for direct response distillation. This paradigm, especially for complex instructions, can degrade the quality of synthesized data, compromising the knowledge distillation process. To this end, our study introduces the Adaptive Modular Response Evolution (AMR-Evol) framework, which employs a two-stage process to refine response distillation. The first stage, modular decomposition, breaks down the direct response into more manageable sub-modules. The second stage, adaptive response evolution, automatically evolves the response with the related function modules. Our experiments with three popular code benchmarks—HumanEval, MBPP, and EvalPlus—attests to the superiority of the AMR-Evol framework over baseline response distillation methods. By comparing with the open-source Code LLMs trained on a similar scale of data, we observed performance enhancements: more than +3.0 points on HumanEval-Plus and +1.0 points on MBPP-Plus, which underscores the effectiveness of our framework. Our codes are available at https://github.com/ChiYeungLaw/AMR-Evol.

## introduction recently, the powerful proprietary large language models (llms), like @xcite , gpt4 @xcite , gemini @xcite and claude @xcite , have showcased impressive code generation ability. especially, gpt4, the most performant model, has recorded * corresponding authors. coding tasks design a function that implements the laplace expansion theorem to calculate the determinant of a 3x3 matrix, but now with the added condition of utilizing nested for loops and ensuring that the matrix elements are limited to integers only. teacher model response def calculate_determinant(matrix): if len(matrix) != 3 or len(matrix[0]) != 3: raise valueerror('matrix must be a 3x3 matrix') @xmath0: @xmath1, 3)] @xmath2. despite their strengths, the closed-source nature sparks accessibility and privacy concerns @xcite . in response, there is a trend of adopting knowledge distillation @xcite to transfer the advanced code generation ability from the proprietary llms to open-source counterparts, thereby enhancing their capabilities while ensuring broader availability and owner autonomy. given that accessing the model weights of proprietary llms is infeasible, the knowledge distillation pipeline is considered as a process where the teacher models synthesize supervised data, primarily consisting of instruction-response pairs @xcite . student models are subsequently trained on this data, enabling the transfer of capabilities from the teacher models. for example, chaudhary (2023) employs the self-instruct method @xcite to prompt the teacher model to generate new coding instructions based on predefined seed tasks. similarly, oss-instruct @xcite utilizes a variety of code snippets sourced from github to inspire gpt-3.5 to produce novel coding instructions. likewise, code evol-instruct @xcite employs iterative prompting to progressively elevate the complexity of code instructions provided by teacher models. each of these methods has proven effective in distilling coding knowledge from teacher models. despite these advancements, there remains an unresolved challenge in enhancing the quality of code response distillation within the data synthesis process. in this setting, code responses serve as labels that teach the student models. previous works have shown that higher-quality responses can lead to more effective distillation @xcite . however, current methods @xcite @xcite tend to rely solely on teacher models for direct response distillation. as shown in figure 1 , this approach is limited by the capabilities of the teacher models, making it difficult to produce accurate responses for complex tasks. the issue becomes even more challenging with methods like code evol-instruct, which deliberately amplify the complexity of instructions. consequently, relying on direct distillation can result in lower-quality responses, ultimately affecting the performance of the student models @xcite . a straightforward yet costly solution to guarantee response quality is to hire human annotators to craft the unit tests for each response. these tests could then be used in an execution-based strategy to validate answers. however, this method is financially prohibitive because it requires the recruitment of annotators with extensive programming expertise. alternatively, depending on the teacher model to automatically generate unit tests for selfrepair @xcite @xcite introduces the same concern of response quality, providing no certainty regarding the correctness of the code repair. to address the challenge of distilling highquality code responses from teacher models, we introduce a novel framework named adaptive modular response evolution (amr-evol). in figure 1 , the example reveals that the direct response distillation can somewhat capture the essential concepts required for solving coding tasks; however, it often deviates from the specific requirements and incorporates logical errors. motivated by this observation, amr-evol leverages the outputs of direct distillation as seed data and employs a two-stage process-namely, modular decomposition and adaptive response evolution-to gradually refine the distilled code responses. by intricately refining the process of response distillation, our framework elicits better knowledge distillation of the student models. in the first stage of our amr-evol, we adopt the idea from modular programming @xcite to manage the complexity of distilling code responses. by utilizing direct responses as the seeds, this method breaks down the coding task into smaller, more manageable sub-modules. this strategy shifts the focus of the teacher models towards solving these sub-modules step-by-step rather than generates a complete solution in a single attempt, whose effectiveness has been verified in recent chain-of-x works @xcite @xcite . additionally, while coding tasks may vary significantly in objectives, the modular components need to construct their solutions frequently exhibit commonalities, or can even be identical @xcite . hence, our adaptive response evolution stage leverages an auxiliary functional module database to store all validated modules for reuse. during response generation, this process utilizes the modules formulated in the decomposition stage to retrieve suitable, pre-validated modules from the database. these related modules serve as in-context examples, aiding the adaptive refinement of responses, thus reducing our sole reliance on teacher models. as evolution progresses, any newly created modules that differ from those in the database are added after a verification process by the teacher model. we apply our amr-evol framework to different student models and select the most representative coding benchmarks, including humaneval (chen et al., 2021), mbpp @xcite , and evalplus @xcite , for evaluation. the results reveal that our amr-evol framework consistently surpasses other response distillation methods, namely direct response distillation, chain-ofthought distillation, and response repairing. these results affirm the superiority of our approach in improving knowledge distillation for llms in code generation. moreover, by integrating our amr-evol with code evol-instruct, one of the sota instruction construction methods, our models achieve better performance than the open-source alternatives trained on a comparable data scale. specifically, we observed an improvement of more than +3.0 on humaneval-plus and +1.0 on mbpp-plus. ## related work llms and code generation. recently, llms have showcased significant achievements across a vast array of tasks. leading tech firms have made substantial progress in developing highly advanced close-source llms, including openai's gpt4 (openai, 2023), google's palm @xcite and gemini @xcite , as well as anthropic's claude @xcite . on the other side, the ai community has also seen the launch of several open-source llms, with model weights becoming publicly available. mistralai has contributed the mistral-series @xcite . google has released ul2-20b @xcite and gemma @xcite . tsinghua university introduced glm-130b @xcite and @xcite , while meta has made available opt @xcite and llama1&2&3 @xcite . furthermore, allen ai has introduced the wholly open-sourced llm, olmo @xcite , and microsoft has released phiseries @xcite . although a gap remains between the open-source models and their closed-source counterparts, this gap is gradually narrowing. in parallel, recent research efforts have been directed towards leveraging llms for coderelated tasks to address the understanding and generation of code. openai has unveiled codex @xcite , google has proposed codegemma (google, 2024), and salesforce has introduced codegen-series @xcite , and codet5&plus @xcite . contributions from tsinghua university include codegeex @xcite , and the bigcode project has developed starcoder1&2 @xcite . meta has also made its mark with the codellama @xcite , while deepseek has open-sourced the deepseekcoder @xcite . these initiatives underscore the growing interest in employing powerful base llms for code generation. our work introduces a novel method for more effectively distilling code knowledge from closedsource models to these open-source base models, thereby enhancing the coding performance. ## method as depicted in figure 2 , we introduce our novel framework, amr-evol, aimed at improving code response distillation to elicit better performance of the student models. in this section, we will provide a detailed discussion of our framework's pipeline. ## conclusion in this study, we present a novel framework, amr-evol, that leverages a two-stage approach-namely, modular decomposition and adaptive response evolution-to enhance code response distillation from teacher models, thereby improving knowledge distillation in code generation. our experiments across three well-known coding benchmarks, hu-maneval, mbpp, and evalplus, demonstrate the effectiveness of our method.