Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLM s (2025)

Abstract: Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development. This paper aims to investigate the fine-tuning of code-generating LLMs using Reinforcement Learning and Direct Preference Optimization, further improving their performance. To achieve this, we enhance the training data for the reward model with the help of symbolic execution techniques, ensuring more comprehensive and objective data. With symbolic execution, we create a custom dataset that better captures the nuances in code evaluation. Our reward models, fine-tuned on this dataset, demonstrate significant improvements over the baseline, CodeRL, in estimating the quality of generated code. Our code-generating LLMs, trained with the help of reward model feedback, achieve similar results compared to the CodeRL benchmark.

## introduction reinforcement learning (rl) has become one of the most powerful llm fine-tuning techniques @xcite . rl integrates feedback into the fine-tuning process, steering the training in the direction of human preferences. there are various approaches to applying rl to llms, but the general idea often consists of three steps: rl has found many applications, one of which being coding assistance @xcite @xcite . according to @xcite , code generation is particularly well-suited for rl because, unlike natural language tasks, the preference data can be created automatically and more objectively through the percentage of passed unit tests. however, the quality of unit test feedback is highly dependent on the test data quality @xcite . when human developers design test cases, they may overlook a path in the control flow graph (cfg) or cover one path multiple times @xcite . these errors may result in biased feedback and, thus, incorrect rl training data. our work aims to evaluate whether symbolic execution improves reward-based fine-tuning of codegenerating models. to achieve this, we enhance the apps dataset @xcite , a realworld coding dataset, by augmenting it with automatically generated test cases created through symbolic execution. this technique executes code with symbolic values @xcite , restricted to specific ranges for each control flow graph (cfg) path, ensuring that every path is covered exactly once. symbolic execution tools analyze the cfg and generate a sample input for every path, eliminating human biases in test case creation. using the augmented apps dataset, we fine-tune the codet5 model @xcite with rl, comparing its performance to codet5-finetuned-coderl @xcite , a codet5 version fine-tuned with rl on the original apps that achieved sota performance on the mbpp benchmark @xcite at the time of its release. finally, we evaluate symbolic execution for direct preference optimization (dpo), a supervised alternative to rl, where the model can be trained directly on a dataset of chosen-rejected code pairs, without the usage of a reward model @xcite . this addition allows us to evaluate the per-formance of symbolic execution under both explicit (rl) and implicit (dpo) reward settings. ## related work there have been invented several frameworks for fine-tuning coding models with rl-based strategies. rltf, reinforcement learning from unit test feedback, utilizes unit test results as multigranular feedback signals that penalize incorrect basic blocks @xcite . ppocoder extends unit test feedback with syntactic and semantic matching scores between generated and ground truth code @xcite . dou et al. (2024) introduce stepcoder, addressing the issue of not penalizing unexecuted code by decomposing generation problems into simple sub-tasks and masking out unreached code. several recent papers introduce systems that combine symbolic execution tools and llms during inference. wang et al. (2024) propose an llm agent that generates execution path constraints for python code by iteratively calling a satisfiability solver. zaharudin et al. (2024) combine llms with symbolic execution tools to identify code vulnerabilities, while chen et al. (2024) apply both to secure medical software. although research has explored rl for fine-tuning code-generating models and integrated symbolic execution with llm inference frameworks, little attention has been paid to combining these approaches. specifically, the use of symbolic execution for fine-tuning code-generating models remains largely unexplored. this paper aims to bridge this gap. ## methodology our approach consists of two main steps: preference dataset creation and llm fine-tuning. first, we use symbolic execution to generate test cases for apps train tasks, produce code solutions, and rank them by performance. we then sample from the ranked codes to train codet5-base @xcite as a reward model, which is subsequently used to optimize the code-generating llm, codet5-large-ntp-py @xcite . ## conclusion in this study, we investigated the intersection of fine-tuning for code-generating models and symbolic execution. by enhancing the apps dataset with symbolic execution inputs, we ensured a solid coverage of paths within the control flow graph. using this enriched dataset, we trained two critic models that significantly outperformed the baseline -the coderl critic. these results indicate the high potential of using symbolic execution tools to generate training data for reward models. the enhanced coverage provided by symbolic execution enabled the reward models to access more informative and accurate training data, thereby improving their ability to evaluate a code's performance. at the same time, while actor and dpo models outperformed their base models, they gained only a slight advantage over the coderl actor. although our critic models predict more precise feedback, the actors stay on a similar level to coderl. we believe that the intersection of reinforcement learning and symbolic execution holds significant potential for advancing code-generating models. future work could investigate the relationship between critic performance and actor effectiveness, optimize hyperparameter configurations for actor training, and explore datasets with further pro-gramming languages or other fine-tuning tasks to achieve similar gains for actor models. with further research, we suggest that symbolic execution combined with reinforcement learning will enable the development of more accurate and robust coding assistants.