Towards Effectively Leveraging Execution Traces for Program Repair with Code LLM s (2025)

Abstract: Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR).However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior.Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces.We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset/model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in promptsand demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently.Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.

## introduction automatic program repair (apr) is a critical challenge in software engineering, aiming to reduce human effort in debugging and fixing software defects. software bugs can lead to significant security vulnerabilities, financial losses, and system failures, necessitating efficient repair mechanisms. while large language models (llms) have demonstrated remarkable capabilities in generating and modifying code, their effectiveness in apr remains constrained by their reliance on static code analysis. recent advancements in knowledge-augmented nlp have emphasized integrating external information into language models to enhance reasoning and accuracy. inspired by this, our research explores augmenting llm-based automated program repair (apr) with program execution traces-structured runtime data that reveal a program's actual behavior. these traces provide diagnostic insights beyond static code analysis. by embedding them into repair prompts, we aim to bridge the gap between static and dynamic program understanding, aligning with trends in knowledge-augmented nlp that leverage external sources to enhance language model capabilities. we frame our work in terms of three research questions (rqs). in rq1 (section 3.2), our objective is to quantify the gains from incorporating execution traces into the apr prompt over the prompts only containing the failing test case as well as the trace-free chain-of-thought prompting baseline @xcite . we find that simply adding the execution trace does not consistently outperform trace-free prompts. to inform a more finegrained approach, in rq2 (section 3.3), we analyze the relationship between trace complexity and the likelihood of the llm producing a working fix. to measure this complexity, we consider two parameters: trace length and the number of variable modifications. we find that the effectiveness of trace-based prompts decreases with the growing length and number of variable assignments. motivated by this finding, in rq3 (section 4), we aim at improving the consistency of trace-based apr by experimenting with three different representations of execution traces: traces in a collated format, llm-optimized traces, and a trace representation conditionally selected based on querying the llm's confidence. we find that llmoptimized trace-based prompts provide the most consistent results with respect to program repair. we additionally perform two follow-up studies: in the first, we compare our trace-based prompting approach with a fine-tuned baseline inspired by tracefixer @xcite ; and in the second one, we directly probe the llm on two trace understanding tasks. the rest of the paper is organized as follows. in section 2, we discuss the related work and how it differs from our approach. section 3 details our methodological setup and covers rq1 and rq2. section 4 covers rq2 and in section 5, we discuss the additional studies. ## related work recent work looked into augmenting code llms with execution information to improve performance on downstream tasks, including apr. selfapr @xcite proposed to use compiler and test diagnostics during self-supervised training of the language model for improving apr. additionally, several works have proposed the use of execution traces for pretraining code llms. in traced @xcite , authors finetuned a bert-like model to predict execution paths and quantized values, which allowed it to outperform an ast-based unixcoder @xcite on clone detection and vulnerability detection. whereas, liu et al.'s program state prediction pre-training improved code search and generation @xcite . finally, tracefixer, based on codet5 and finetuned with execution traces, showed a 13% improvement in apr on synthetic bugs over the codeonly baseline but struggled with real bugs, hinting at potential generalization limitations @xcite . among training-free approaches, self-debug @xcite improved program generation by generating code explanations directly from the llm, in a chain-of-thought fashion, as part of solving the apr task. to the best of our knowledge, all of these works do not consider the effect of putting execution traces in the prompt of a pretrained llm. ## analyzing the impact of execution traces on program repair in this section, we analyze the effects of adding traces in the llm prompt on apr performance, compared to two trace-free baselines @xcite , @xcite . additionally, we perform a differentiated analysis of apr performance based on trace complexity. we formulate the corresponding two research questions as follows. rq1. are prompts with execution traces more effective at program repair than prompts without traces? rq2. how does trace complexity affect the effectiveness of trace-based prompts? we selected three apr datasets: refactory @xcite , runbugrun (prenner and robbes, 2023), and humaneval-java @xcite . refactory includes nearly 2000 faulty python programs submitted by students, enabling coverage of diverse mistakes. runbugrun, derived from co-denet, contains a quarter million submissions for 4000 distinct problems; we sampled 1000 python bugs for evaluation. humaneval, originally for python, was adapted into humaneval-java, injecting synthetic bugs for apr testing. each dataset includes at least 5 test cases per problem. for runbugrun, we implemented a wrapper to handle input/output via standard input and print statements for accurate result comparison. models. with the landscape of state-of-the-art code llms rapidly changing, we chose use two most widely studied commercial models from openai for ease of comparison with other work: gpt-3.5 turbo @xcite and @xcite . these two models represent two different performance tiers both in terms of the number of parameters and different release timelines, hence, studying these models could shine the light on the llms' evolving ability to reason about program execution across product generations. while there is undoubtedly scope for including more proprietary as well as open source models, given our narrow focus on traces, we leave this to be explored in future work. execution traces generation. as the program is being executed, it is possible to step through it programmatically, while also capturing every change to the function's variables, akin to interactive debugging. pysnooper(pys) library for python provides this functionality via a decorator that can be added to a function of interest to automatically log state changes, such as variable initialization and modification, subroutine calls, returned values, and runtime exceptions. crucially, each state change reference a specific line of code on which it occurred. examples of execution traces are given in the appendix a.2. before appending traces to the prompt we perform basic postprocessing, including the removal of timestamps and stripping of terminal formatting command sequences. prompt types. we follow the instruction template for complete function generation used by @xcite , expanding it with two additional types of information, namely, a failing test case (henceforth, referred to as error prompts) and a program execution trace (referred as trace prompts). we offer our rationale for these choices, along with other prompt types considered, in appendix b. to ensure the prompt and response fit within the gpt model context size, we truncate the content of the prompt if the number of lines exceeds 200. we have added an example of all the prompts in section a in the appendix. baseline. we consider the prompt-based baseline self-debug. with this baseline, we explore prompting llms using execution traces generated by llms themselves (instead of actual program execution traces). this baseline inspired by self-debugging @xcite where llms are prompted to debug their own generated code. in particular, we draw inspirations from the explanation step of this work where the model is asked to generate execution traces for a predicted code. we tailored self-debugging's prompts to fit our usecase: in our prompts, we provide llms with a program and a test case feedback, and ask them to trace through the execution of the program and determine the needed fix, and correct the function accordingly. we perform these experiments with both gpt-3.5 and gpt-4. metrics. in previous work on apr, models are evaluated either at the granularity of distinct bugs solved @xcite or individual test cases passed @xcite . in contrast, we generate multiple prompts for each program tailored to a specific failing test case and its corresponding execution trace (e.g., a.1, a.2). rather than aggregating predictions from multiple samples, we generate a single prediction per test case-specific prompt and aggregate across prompts when computing metrics. the key metrics are correct fix accuracy (cfa), the percentage of fixes passing all test cases, and correct program accuracy (cpa), the percentage of programs with at least one correct fix. we do not report test caselevel accuracy, as it can be too lenient and doesn't account for variations in the number of test cases per program. 3.2 rq1. are prompts with execution traces more effective at program repair than prompts without traces? in this research question, our objective is to evaluate the effectiveness of including program execution traces into llm prompts, for solving apr tasks, compared to the baselines. the effectiveness is measured through reporting cpa and cfa. the evaluation results can be found in on the humaneval-java dataset with gpt-3.5, error-only prompts are only ahead of trace-based prompts in terms of fix accuracy but are slightly behind in program accuracy. meanwhile, with gpt-4, trace-based prompts consistently outperform error-only prompts on both metrics. results on runbugrun paint a similar picture, where gpt-3.5 doesn't seem to benefit from including traces, while gpt-4 gets a tangible lift over the error-only prompts. overall, on two out of three datasets, trace-based prompts significantly improve the ability of gpt-4 to generate working bug fixes. while gpt-3.5 lagging behind in terms of absolute scores irrespective of prompt type is expected, more broadly, its inability to benefit from execution traces (even degraded performance) could highlight a qualitative generational gap when it comes to emergent abilities of llms. notwithstanding, there remain a few unexplained results, such as the lack of performance gain from using traces on the refactory dataset and the unusually strong performance of the self-debug baseline in one particular configuration. to gain a fine-grained understanding, in the next research question, we focus on studying the varying complexity of execution traces and how they affect downstream apr performance. ## impact of modified traces as we find that longer trace length could have a negative impact in the effectiveness of gpt models, we focus on modified trace strategies and their impact on the effectiveness of the model. in this section, we discuss one research question. rq3. can the format of traces be optimized to guarantee gains for apr? ## conclusion in this study, we examined the impact of incorporating program execution traces into prompts on the program repair capabilities of the gpt model family. our findings indicate that trace-based prompts do not consistently outperform error-based prompts; their effectiveness varies with the dataset and llm used. analysis reveals that longer traces and more variable assignments reduce prompt effectiveness. using this insight, we developed variations of trace-based prompts, finding that llmoptimized traces offer more consistent improvements without limiting trace complexity heuristically. we validated our results against a fine-tuned baseline and found that llms have limited capacity for trace generation, explaining the weaker performance of the self-debug baseline and highlighting the potential utility of traces in code tasks.