The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python (2023)

Abstract: Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming. Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size. Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.

## introduction pretrained large language models (llms) are rapidly becoming one of the dominant paradigm for large variety of language tasks @xcite , including programming code generation and completion @xcite . llms have demonstrated increasing performance with increasing model size 1 on many practical tasks @xcite including programming tasks @xcite , recently, however, researchers len, @xmath0, len def print_len(x): tasks with inverse scaling generally either involve social biases @xcite , where the larger models (arguably correctly) learn undesirable biases from biased training sets, or involve examples of natural language that are highly atypical but still easily understandable by a human @xcite . these tasks may involve unusual discourse pragmatics or they may require reasoning about counterfactual knowledge, however, since they tend to be highly artificial, it could perhaps be argued that they are edge cases which may not represent serious failure modes for practical applications. in this paper we present a novel type of inverse scaling task involving python code generation under a redefinition of default identifiers. this has both practical implications (redefinition of default identifiers is a metaprogramming technique used in popular libraries), and broader scientific implications, as it shows that llms fail to reason about the deep, abstract semantic structure of programming languages, and these flaws are not ameliorated, but in fact may be even worsened, by increasing model size. crawl github extract functions swap builtins heads (prompts) original bodies (bad classes) corrected bodies (good classes) defined syntax and semantics which makes them especially suited to automatic analysis and procedural generation. they are scientifically interesting because they can be used for automatic generation of examples of coding problems and their evaluation against an objective ground truth, whereas most nlp tasks have enough ambiguity that require human annotation in order to produce high-quality examples. furthermore, this research is also of practical importance for software engineering tools that use llms, such as github copilot, foot_0 which are starting to be widely adopted by developers. ## methodology we describe the motivation behind our task ( §2.1) and the task itself ( §2.2), followed by the way we collected the data for the task ( §2.3). we release our dataset as well as the code used to generate it and replicate our experiments foot_1 . ## experiments we next describe our experiments with a likelihood calculation of correct and incorrect completions ( §3.1) and chat llms ( §3.2), and then present a qualitative analysis ( §3.3). ## related work recent work sought to characterize the quality of llms on a variety of tasks: big-bench (srivastava et al., 2022) is a large collaboration which resulted in a suite of hard, disparate tasks which were used to evaluate various llms. the study found that scaling can be slower and less smooth than expected by naive scaling laws, and social biases sometimes show inverse scaling, also observed by @xcite . @xcite investigated the effect of example selection in fewshot learning for llms, finding that previous studies generally overestimated model quality due to methodological issues. @xcite attempted to measure the truthfulness of the answer provided by llms on tasks involving real-world knowledge, finding that while larger models tend to provide more informative answers, they also tend to be less truthful. however, this effect might be confounded due to the dataset design to specifically be adversarial for the largest model being evaluated @xcite . @xcite showed that similar to our case, mathematical article processing is sensitive to semi-invariant symbol replacements. @xcite provide a broad survey about hallucination (generation of fluent yet incorrect information) by natural language generation models. ## conclusions we explored the ability of large language models to predict the correct continuations of fragments of python programs in scenarios where the correct continuations are statistically uncommon due to the redefinition of identifiers caused by a statement that we included in the prompt. not only all the tested models fail at this task, but some model families even display inverse scaling: they become worse, rather than better, with increasing model size. these results suggest that llms rely on "shortcut learning", i.e., weak, unstable, mostly lexical correlations in the data, rather than an understanding of the semantics of the data (in this case, python code) at a deep level. we believe that our results are important both for a better scientific understanding of the capabilities of llms and for their practical relevance as a core technology for automated code generation tools. future work could investigate scaling effects at larger model sizes, as well as on other programming languages.