When Benchmarks Talk: Re-Evaluating Code LLM s with Interactive Feedback (2025)

Abstract: Programming with a coding assistant is a fundamentally interactive process, yet existing static benchmarks fail to capture key features of model-user collaboration. We introduce an interactive evaluation pipeline to examine how LLMs incorporate different types of feedback in a collaborative setting, in which we obfuscate the input of static coding benchmarks so that the code model must interact with a simulated user. Across 10 models and 3 datasets, the relative rankings of models often permute greatly between static and interactive settings, despite models being fairly robust to feedback that contains errors. We also observe that similarly effective feedback types differ in terms of how models respond to higher- vs. lower-quality feedback. Moreover, feedback type impacts the degree to which the models make aesthetic or behavioral edits to their output. Our work aims to “re-evaluate” model coding capabilities through an interactive lens toward bridging the gap between existing evaluations and real-world usage.

## introduction programming with a language model is a highly collaborative process, where developers interact with code models to provide updated information about initially underspecified requests or critique the output of the code model. thus, giving and receiving feedback are critical elements of the process in which programmers use code models @xcite . for example, chat interfaces like @xcite or the chat panel of github copilot @xcite facilitate multi-turn conversations in which programmers can iteratively refine a piece of code by providing * equal contribution. despite the popularity of these tools, existing static benchmarks that measure task performance often rely on a simple input-output configuration, where the question is well defined and the model is asked to generate the whole completion in one shot @xcite @xcite . while this relatively simplistic setting is scalable and enables effiwe obfuscate the input of existing fully specified datasets to reflect how programmers tend to underspecify requests to llms (e.g., via docstrings or comments) in practice. (b) as developers may interact with models in a variety of ways, we explore 4 different feedback types and introduce a pipeline that mimics the iterative refinement loop that programmers often use with chat models, (c) where the code model generates a solution using feedback on its previous solution, (d) and the user provides updated feedback to the code model. building evaluations of programming assistants that more closely mimic this setting enables a better understanding of model behavior and potential pitfalls in the interactive setting, such as model capability with respect to processing feedback, the effects of different feedback types, or model robustness. we bridge the gap between existing evaluations and real-world use cases by benchmarking how different feedback types impact model behavior in a simulated interactive setting @xcite . we propose an evaluation method that transforms a static coding benchmark into an interactive, collaborative one (figure 2 ). the pipeline components include input obfuscation to create underspecified problems to induce collaboration, a simulated user for scalability, and multiple types of feedback (code feedback, query rephras-ing, paragraph, and sentence). across 10 coding models (6 open and 4 closed) and 3 coding benchmarks, we find that relative performance between models often changes between static and interactive settings (figure 1 , bottom), suggesting that models perform differently in the static vs. interactive settings. beyond performancebased metrics, we analyze important components of model-user interactions, including feedback quality and code model steerability. we use these insights to investigate the effects of different feedback types. for example, we find that paragraph and code feedback tend to lead to the highest performance boost compared to other feedback types (e.g., sentence or query rephrasing). however, when considering the effect of feedback quality on performance, we find that unlike para-graph, code feedback's higher-quality feedback makes output worse more frequently than its lower-quality counterpart for stronger models. furthermore, paragraph leads to more surface-level edits than code feedback, whereas users may prefer variation in model behavior to be robust to changes in feedback type. our work provides a new approach to investigating the downstream ramifications of different interactive programming settings and their effects on model behavior. we open-source our evaluation pipeline 1 , which makes it easy to add static benchmarks and turn them into interactive ones, to facilitate the evaluation of more models and datasets. ## methodology figure 2 provides an overview of our pipeline for interactive evalution, which is driven by an iterative refinement loop in which the user model and code model interact. at each step of the loop, the code model is given the obfuscated programming question in natural language, the code from the previous attempt, and the user model's feedback on the previous attempt. each iterative refinement loop lasts 5 steps for each question, terminating early if a correct solution is reached in less than 5 steps. ## static vs. interactive performance we compare the performance of models in static and interactive evaluation settings. to measure the effectiveness of different feedback types, we compare against the static and the self-critique baseline settings. the static setting of each dataset evaluates the code model on the original, fully specified questions. in this setting, the code model is not given any feedback and the first attempt is used as the final output. to match the test-time compute of the interactive settings, the self-critique baseline setting uses five iterations of self-critique to generate feedback using the underspecified question and the output from the previous step @xcite . for this setting, no additional information of the original question or solution is given and no user is involved. ## feedback quality to understand whether the ranking changes in section 3 are due to variations in feedback quality across models, we develop a proxy for feedback quality and examine its effect on how code models interact with feedback. ## model steerability we extend our analysis beyond performance to investigate steerability, or how much a code model adjusts its previous solution in response to feedback. we show that drops in performance in interactive settings are likely due to ineffectively incorporating feedback, rather than outright ignoring feedback in the next iteration of the solution. ## conclusion we propose a new approach to evaluating code models by introducing an interactive pipeline where the code model must collaborate with a simulated user to solve underspecified coding problems with different feedback types. we find that the relative performances of models change radically between static and interactive settings. we analyze key elements of model-user interactions, such as feedback quality and model steerability, to provide insights into the downstream effects of feedback type on model behavior and feedback effectiveness. our work bridges the gap between existing static benchmarks and real-world usage, and we hope to inspire future work on scalable methods for evaluating models in a collaborative setting.