Focused- DPO : Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points (2025)

Abstract: Code generation models have shown significant potential for automating programming tasks. However, the challenge of generating accurate and reliable code persists due to the highly complex and long-reasoning nature of the task. Even state-of-the-art models often fail in code generation due to small errors, which can drastically affect the overall functionality of code. Our study identifies that current models tend to produce errors concentrated at specific error-prone points, which significantly impacts the accuracy of the generated code. To address this issue, we introduce Focused-DPO, a framework that enhances code generation by directing preference optimization towards these critical error-prone areas. This approach builds on Direct Preference Optimization, emphasizing accuracy in parts prone to errors. Additionally, we develop a method called Error-Point Identification, which constructs a dataset that targets these problematic points without requiring costly human annotations. Our experiments on benchmarks such as HumanEval(+), MBPP(+), and LiveCodeBench demonstrate that Focused-DPO significantly improves the precision and reliability of code generation, reducing common errors and enhancing overall code quality. By focusing on error-prone points, Focused-DPO advances the accuracy and functionality of model-generated code.

## introduction code generation has emerged as a pivotal task in artificial intelligence, enabling models to automate essential software development tasks. code models @xcite @xcite have demonstrated remarkable capabilities in code generation tasks. these advancements have significantly improved developer's productivity, accelerating software delivery timelines. despite their success, generating correct code remains a substantial challenge due to the com-* ge li and zhi jin are the corresponding authors. figure 1 : error-prone points in generated code from qwen-2.5-coder-instruct-7b. we sample 20 outputs for this question. outputs have common prefixes and suffixes, differing mainly at yellow-highlighted error points. continuing generation at these points leads to drastically different accuracies (90.02% vs. 3.17%). this disparity is not seen in non-highlighted parts. when examining the outputs of current code generation models, we find that errors are not evenly spread across the code. large language models tend to produce errors concentrated in certain errorprone points, even when sampling multiple times with a high temperature. we illustrate this phenomenon in figure 1 , which shows error-prone points highlighted in yellow. despite the overall code having similar prefixes and suffixes, differences at these highlighted error points significantly impact the final code accuracy. generating code from correct outputs at these error-prone points can achieve a final accuracy of up to 90.02%, whereas starting from incorrect outputs reduces accuracy to 3.17%. parts of the code, such as function headers (usually at the prefix) or return statements (usually at the suffix), often follow familiar patterns. however, some middle parts of the code, which involve more complex reasoning, are more prone to errors. errors in these parts can disrupt and affect the entire program's reliability. it is crucial to address these error-prone points for code generation. however, existing studies on code generation overlook this problem. while standard training approaches such as supervised fine-tuning (sft) @xcite help improve overall output quality, they do not specifically focus on the crucial parts necessary for correctness. methods like direct preference optimization (dpo) @xcite aim to align outputs with preferences (e.g., "chosen" vs. "rejected"), but often overlook fine-grained error-prone points of the code. as a result, these trained models might generate code that appears correct initially but contains critical issues at the error-prone points, ultimately affecting overall accuracy. to tackle these issues, we introduce focused-dpo, a framework designed to enhance code generation through focusing preference optimization on error-prone points. focused-dpo builds on direct preference optimization by emphasizing accuracy improvement in areas where errors are most likely to occur. unlike traditional methods that treat all parts of the code equally, focused-dpo specifically targets those error-prone points, which are essential for the overall correctness of the program. focused-dpo is a data-driven preference optimization method that relies on a specially constructed dataset with identified error-prone points. we propose a dataset construction method named error-point identification, which includes an automated pipeline to construct paired code preference datasets. this method extracts concepts from real code repositories and synthesizes programming problems. by concurrently generating code and tests, and using a page-rank-inspired algorithm for ranking, we determine the relative performance of all generated code. error-point identification employs common prefix and suffix matching to precisely locate error-prone points. additionally, our method automatically identifies error-prone code parts, eliminating the need for costly human input, making it scalable and efficient for a variety of programming tasks. we evaluate focused-dpo using standard benchmarks such as humaneval(+) @xcite , mbpp(+), and livecodebench @xcite , and observe significant improvements over existing methods. even for models like qwen2.5-coder, which already have undergone large-scale alignment training, focused-dpo still achieves a 42.86% relative improvement on extremely hard competition-level problems in livecodebench. the results show notable increases in the generation quality on error-prone points, highlighting focused-dpo's effectiveness in enhancing the accuracy of code generation. our contributions are summarized as follows: ## related work large language models (llms) have made significant progress in generating code from natural language descriptions, showing great potential for automating software development tasks @xcite . models @xcite @xcite @xcite have demonstrated strong performance, thanks to extensive training on diverse datasets. to further enhance their capabilities, posting training methods like supervised fine-tuning (sft) @xcite @xcite and direct preference optimization @xcite @xcite @xcite are commonly applied. preference optimization approaches focus on aligning model outputs with desired outcomes by prioritizing more favorable responses over less favorable ones. however, existing dpo approaches fail to address one important issue: they do not directly target the most error-prone points in generated code. errors in these high-impact parts can lead to significant quality and reliability issues in the final output. we aim to address this issue by focusing the preference optimization learning on these error-prone points in the generated code. some fine-grained preference optimization methods @xcite @xcite @xcite have shown strong potential in domains like mathematics, which rely heavily on natural language reasoning. step-dpo @xcite and step-controlled dpo @xcite propose generating step-wise preference datasets to enable optimization learning based on the standard dpo loss. tdpo @xcite enhances the dpo loss by incorporating forward kl divergence constraints at the token level, achieving fine-grained alignment for each token. cdpo @xcite proposes a tricky method to find the critical token in the thought chain that affects overall accuracy. however, the identified tokens are typical in natural language and the method does not apply to code, which features similar overall patterns but relies on specific key elements in long reasoning processes. however, in the context of code generation, where a small error-prone point can lead to major functional errors, these exisiting methods often struggle to construct adequate datasets or fail to achieve ideal improvements due to weak fine-grained reward signals. to address this, we propose focused-dpo, a framework that improves code generation by focusing on optimizing these high-impact parts. our dataset construction method employs a self-generation and validation process to construct datasets that explicitly identify errorprone points, ensuring the optimization learning process directly enhances the parts of the code that matter most for overall correctness. ## experiment setup we aim to answer the following research questions: rq1: are there error-prone points in generated code that significantly affect the correctness of the output? this question addresses the core motivation behind focused-dpo. to investigate this, we construct the validation dataset following section 3.2. this setup provides empirical evidence supporting the theoretical underpinnings of our focused-dpo. rq2: can focused-dpo improve the generation quality of code models, even those that have already been heavily post-trained with alignment techniques such as standard dpo? to explore this, we evaluate focused-dpo on several widely-used code generation benchmarks, including humaneval @xcite , humaneval+ @xcite , mbpp @xcite , mbpp+, and livecodebench @xcite . rq3: how do different components of the focused-dpo loss formulation affect model performance? ablation studies include evaluating our dataset construction method, as well as key components in our loss formulation. ## discussions 6.1 error-prone points identification in our focused-dpo method, we introduce a dataset construction technique called error prone identification to automatically identify error-prone points in generated code. to assess the correctness of the code, we employ a self-generation-andvalidation mechanism based on pagerank, which captures the relative quality of different code snippets @xcite . we are not like approaches such as magicoder @xcite , which directly use all test cases as ground truth. in our experiments we use the policy model to generate datasets. since the policy model's generation quality is not as robust as that of more powerful models like gpt-4 (used in magicoder), the pagerank-based method allows us to automatically filter out lower-quality test cases (those with lower scores after iteration), thereby ensuring higher overall dataset quality. we find that different models exhibit varying levels of accuracy across different problems. therefore, for each model's training dataset, we performed necessary filtering by removing code problems with excessively high or low accuracy rates, ensuring a consistent number of code problems in the final dataset. moreover, we observe that models tend to exhibit similarities in error-prone points when solving the same problems. for example, when comparing the error-prone points identified by deepseekcoder-instruct-6.7b and qwen2.5-coder-instruct-7b models on the same set of programming problems, we found a 32% overlap. this indicates that there are commonalities in the errorprone points across different models. in our ablation studies, we compare error-prone points constructed using the git-diff method and the step-dpo method, noting slight differences in the final results. balancing effectiveness and efficiency, we use the method based on prefix and suffix, which allows us to identify error-prone points in generated code in a simple yet effective manner. we plan to further explore more identification strategies in future work. ## conclusion we propose focused-dpo, a framework that improves code generation by focusing on error-prone points. these critical parts significantly impact overall program correctness. focused-dpo improves direct preference optimization by prioritizing these points, using our error-point identification method to create datasets without costly human annotations. evaluations show focused-dpo reduces errors and improves quality, even in heavily post-trained models. this research highlights benefits of focusing on fine-grained preference optimization in ai-driven software development. @xcite . these online rl alignment techniques typically require substantial training resources, high-quality datasets, and complex reward environments, making their application highly resource-intensive. in contrast, offline alignment methods such as focused-dpo approximate similar optimization objectives while introducing necessary simplifications and derivations. this allows focused-dpo to achieve comparable or even equivalent optimization results with significantly lower resource requirements. moreover, we leverage prior knowledge discovered in this work: the insight that only a small part of the generated code-specifically, the error-prone points-plays a critical role in determining the overall correctness of the output. by incorporating this insight into the training loss design, we further enhance training efficiency and effectiveness. focused-dpo's low resource requirements and reliable performance make it applicable to a wide range of code generation scenarios. further exploration of how focused-dpo compares to these advanced rl techniques in performance and efficiency remains an area for future investigation.