Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models (2025)

Abstract: Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize and reason effectively. These challenges underscore the necessity for more systematic approaches to enhance the reasoning capabilities of Code LLMs. To address these issues, we introduce a generic framework to support integrating semantic information (e.g., execution trace) to code task-relevant prompts, and conduct a comprehensive study to explore the role of semantic information in enhancing the reasoning ability of Code LLMs accordingly. Specifically, we focus on investigating the usefulness of trace-based semantic information in boosting supervised fine-tuning(SFT) and post-phase inference of Code LLMs. The experimental results surprisingly disagree with previous works and demonstrate that semantic information has limited usefulness for SFT and test time scaling of Code LLM.

## introduction code large language models (code llms) have emerged as prominent programming assistants, demonstrating remarkable performance across various coding tasks, including program repair @xcite , code generation @xcite , and code summarization @xcite . recently, several code llms have been introduced, each characterized by distinct training schemes. for instance, llama3.1 @xcite , is fine-tuned with code infilling tasks and long code input contexts, complemented by an instruction fine-tuning process. similarly, @xcite is trained on more than 2 trillion tokens using a fill-in-the-blank task to enhance its code generation capabilities. these models focus on learning contextual information from code and docstrings, advancing their general understanding of code @xcite @xcite . however, these approaches predominantly capture the static dimensions of code (e.g., tokens and context), while neglecting the dynamic semantics crucial for a comprehensive understanding of code. recent studies have highlighted this limitation, revealing that even state-of-the-art models like gpt-4 struggle to reason about runtime behaviors of code @xcite . understanding code semantics and accurately predicting runtime behavior is critical, particularly for practical coding applications that require semantic understanding @xcite . this underscores the urgent need to investigate methodologies that enhance the reasoning capabilities of code llms for better semantic understanding, such as coverage prediction @xcite and output prediction @xcite abilities. to address these challenges, two primary methods have been proposed to bolster the reasoning capabilities of code llms in code generation tasks: 1) inference with semantic information -integrating feedback, such as error information to the inputs (or outputs) at inference time, and 2) finetuning with semantic information -directly enhancing the models' intrinsic reasoning by integrating semantically enriched fine-tuning data, thereby enabling improved predictions @xcite . in this work, we consider both approaches and explore whether and how incorporating code semantic information can enhance the performance of code llms. specifically, we investigate evaluate their effectiveness across various code-related tasks. the core challenge lies in identifying and collecting appropriate semantic data for training or inference to improve the semantic comprehension of code llms. recent efforts @xcite have begun to address this by finetuning models using dynamic data, such as execution behaviors, to enhance semantic understanding. while these approaches have shown promise, they employ diverse semantic representations, such as natural language descriptions of programs or execution traces. there remains a lack of 1) a unified study and tool that supports all semantic representations, and 2) systematic understanding regarding how different training data compositions, particularly in terms of semantic representations, impact code reasoning and generation capabilities. to this end, we propose and implement a generic framework that facilitates the generation of multiple types of semantic representations, supporting post-training, one-time inference, and the scaling of test-time computation during inference. based on this framework, we conduct a systematic study to explore the efficacy of semantic information in boosting code generation. specifically, we integrate different semantic representations (i.e., different code execution traces) into the input data (prompt) during both sft time and inference time to assess their impact. additionally, we examine the influence of different training strategies (e.g., parameter-efficient fine-tuning) on the effectiveness of semantic information. different from previous research findings @xcite , our experimental results demonstrate that integrating existing semantic information into the input prompt has limited benefits to the performance of tuned code llms. to summarize, the main contributions of this paper are: ## problem statement we address the problem of enhancing the code generation capabilities of code llms by integrating code semantic information into the input prompts. current code llms primarily rely on static text data, which often fails to capture the nuanced semantics crucial for thorough code understanding @xcite @xcite . inspired by the practices of human developers, who iteratively refine code through reasoning and semantic assessment rather than relying solely on dynamic testing or runtime feedback, we note that such reasoning and refinement processes are largely absent in existing models @xcite . there are two main sub-problems we are interested in for code llms, 1) fine-tuning with semantic information, and 2) inference with semantic information. here, θ may control how many refinement steps to run, which candidate paths to verify, subject to the budget n . verification signals (such as code execution) enable the model to discard incorrect paths or improve partially correct ones, while iterative refinement uses feedback to converge on better outputs. by strategically allocating test-time resources, a trace-based verify-and-refine loop can substantially boost accuracy without additional training. ## experiment design based on our framework, we conduct a comprehensive study to explore the usefulness of code semantic information for code llms during fine-tuning and test-time scaling, respectively. specifically, for the fine-tuning part, we utilize our constructed datasets to fine-tune code llms first, and then evaluate their capabilities on different programming tasks using the basic evaluation paradigm. for the test-time scaling evaluation, we employ different test-time scaling strategies to help assess the ability of fine-tuned code llms from the previous step to investigate the usefulness of semantic information. datasets. ## conclusion this paper introduces a generic framework for generating trace-based code semantic information. based on this framework, we systematically evaluate the usefulness of trace-based code semantic information for fine-tuning and inference of code llms. the results highlight that existing code semantic information does not enhance fine-tuning and test-time scaling of code llms. this work can serve as the new baseline for the study of leveraging semantic information to enhance code llms. this opens up several avenues for future research. first, it is essential to design new forms of semantic representations that are more aligned with how code llms process and understand code, potentially incorporating higherlevel abstractions or contextual cues. second, future work should explore more effective strategies for integrating semantic information into model training and inference pipelines -such as architectural modifications, specialized pretraining objectives, or more adaptive prompting techniques.