The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations? (2024)

Abstract: While language models are increasingly more proficient at code generation, they still frequently generate incorrect programs. Many of these programs are obviously wrong, but others are more subtle and pass weaker correctness checks such as being able to compile. In this work, we focus on these counterfeit samples: programs sampled from a language model that 1) have a high enough log-probability to be generated at a moderate temperature and 2) pass weak correctness checks. Overall, we discover that most models have a very shallow understanding of counterfeits through three clear failure modes. First, models mistakenly classify them as correct. Second, models are worse at reasoning about the execution behaviour of counterfeits and often predict their execution results as if they were correct. Third, when asking models to fix counterfeits, the likelihood of a model successfully repairing a counterfeit is often even lower than that of sampling a correct program from scratch. Counterfeits also have very unexpected properties: first, counterfeit programs for problems that are easier for a model to solve are not necessarily easier to detect and only slightly easier to execute and repair. Second, counterfeits from a given model are just as confusing to the model itself as they are to other models. Finally, both strong and weak models are able to generate counterfeit samples that equally challenge all models. In light of our findings, we recommend that care and caution be taken when relying on models to understand their own samples, especially when no external feedback is incorporated.

## introduction in the past year, language models such as code llama @xcite , @xcite , and @xcite have demonstrated great advances in code generation. their success has primarily been due to their strong code generation abilities, as measured by benchmarks such as humaneval @xcite and mbpp @xcite as well as their usefulness in general-purpose code writing. while these models are able to produce correct code for impressively complex specifications, they just as often produce incorrect code. some of these incorrect programs contain egregious mistakes, but others fail in more subtle ways. we focus our attention towards the second group, which we call counterfeit samples. we define a counterfeit sample to be a program sampled from a code language model which is 1) good enough to be generated by the language model at a moderate temperature and 2) pass weak but nontrivial correctness checks. in this work, we study the extent to which models can understand these counterfeit programs. the second criterion of passing nontrivial correctness checks distinguishes programs with more subtle errors from those that trivially fail and are likely uninteresting. in fig. 1 , we show an example of an incorrect, counterfeit, and correct program. because we use relatively weak correctness checks, many counterfeit programs can still be easily detected as wrong by a human. we provide empirical evidence that code language models have a shallow understanding of these counterfeit samples (sec. 3) via three evaluations: correctness checking, execution prediction, and program repair. for correctness checking, the model is asked to assess whether a short piece of code correctly implements a natural language specification (sometimes with test cases). for execution prediction, the model is given a program-input pair and asked to predict the output of executing the program on the given input. for fairness, we ensure the programs are generally short and that execution does not require complex calculations. for repair, the model is given the counterfeit program alongside its original specification and is asked to correct it. first, we find that models frequently misjudge counterfeit samples as correct. second, models are much worse at reasoning about the execution of counterfeits than their correct companions, often executing counterfeits as if their semantics matched those of a correct program. third, models falter at repair: the likelihood of a model successfully repairing a counterfeit example is often even lower than that of generating a correct program when sampling from scratch. through further analysis, we find that counterfeit samples have other unexpected properties (sec. 4). we find, for example, that counterfeit samples from problems that are easier for the model to solve are not easier to assess and only slightly easier to execute and repair, highlighting an inconsistency between generation and understanding capabilities. we also observe that models don't perceive their own counterfeit samples differently from other models' counterfeits and that models of all capability levels are able to generate equally difficult counterfeit samples. overall, we find that these counterfeit samples are, in a sense, adversarial to the model: models often struggle to assess their correctness, reason about their execution as if they were correct programs, and repair them at a low rate. understanding counterfeit samples is a prerequisite to many downstream applications in which models use their own feedback to improve themselves. therefore, in light of our findings, we recommend exercising caution in these schemes such as self-repair and model-based reranking of outputs, especially when no external feedback is incorporated. ## qualitative analysis in general, most counterfeit programs fall into one of three broad categories: (1) error in algorithmic design or implementation, (2) incorrectly understanding or completely ignoring details in the specification, (3) failing to address corner cases in the input space. examples of each are provided in appendix e. ## conclusion in this work, we bring attention to the counterfeit samples of a code language model: incorrect programs that a model thinks are correct and can pass surface-level correctness checks. we observe that in a sense, these counterfeit samples are adversarial to the model: models often cannot assess their correctness, reason about their execution, and struggle to repair them. compared to other models, gpt-4 may be different from other evaluated models in this regard, in that they are much less susceptible to the traps we observe on counterfeit samples from other models. while we operate in the domain of code, where it is simple to precisely check a model's understanding, we suspect that the same phenomena occur more generally in language models, which is consistent with the findings from @xcite . because models being able to understand their own counterfeit samples is a prerequisite to strong selfrepair and self-verification schemes, we recommend that others be critical and careful in light of our findings.