Grammar-Based Code Representation: Is It a Worthy Pursuit for LLM s? (2025)

Abstract: Grammar serves as a cornerstone in programming languages and software engineering, providing frameworks to define the syntactic space and program structure. Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance. However, as language models scale to the billion level or beyond, syntax-level errors become rare, making it unclear whether grammar information still provides performance benefits. To explore this, we develop a series of billion-scale GrammarCoder models, incorporating grammar rules in the code generation process. Experiments on HumanEval (+) and MBPP (+) demonstrate a notable improvement in code generation accuracy. Further analysis shows that grammar-based representations enhance LLMs’ ability to discern subtle code differences, reducing semantic errors caused by minor variations. These findings suggest that grammar-based code representations remain valuable even in billion-scale models, not only by maintaining syntax correctness but also by improving semantic differentiation.

## introduction context-free grammars are the fundamental way to specify the syntactic space of a programming language, and with the grammar specified, a program can be parsed into a syntax tree, revealing its structure @xcite . building on this foundation, leveraging grammatical knowledge (e.g., grammar rules) to pre-train large language models (llms) has emerged as a promising strategy for code-related tasks, such as code generation @xcite @xcite . existing research has explored grammar-based code representation @xcite ., † work done during internship at kuaishou technology. *corresponding authors: zeyu sun and yingfei xiong. however, as language models scale to the billionparameter level and beyond, extensive pre-training on large code datasets enables them to implicitly learn syntax rules, making syntax errors increasingly rare @xcite @xcite . for example, even 1b scale models, such as @xcite and qwen2.5 @xcite , achieve high accuracy in code generation, consistently producing syntactically valid code. this phenomenon suggests that large models are able to understand the structure of the program and raises a critical question: is grammar-based code representation still beneficial for billion-scale llms? to answer this question, we conduct an experiment comparing grammar-based representation and token-based representation approaches on 1.3b, 1.5b, and 7b parameter models, respectively. the results demonstrate that grammarbased models (i.e., grammarcoder-1.3b-base, grammarcoder-1.5b-base, grammarcoder-7b-base) significantly outperform token-based models, even though token-based models rarely make syntax errors. for example, on the mbpp dataset @xcite , grammarcoder-1.3b-base achieves an almost seven percentage point improvement in pass@1 compared to deepseek-coder-1.3b-base trained on the same data. this suggests that grammar rules enhance code generation beyond syntax correction, even in billion-scale models. the result leads to a second question: why do grammar-based models improve performance if token-based models already produce syntactically correct code? to investigate this question, we examine the differences between grammar-based and token-based code representations. our analysis reveals that minor token-level modifications can lead to substantial semantic shifts, rendering correct programs incorrect. in contrast, while these subtle variations may appear insignificant at the token level, they often map to clear structural differences in grammarbased representations, enabling the model to distinguish more effectively between correct and incorrect code. experimental results further confirm a correlation between higher performance and the ability of grammar-based code representation to amplify representational differences for semantic shifts, indicating that grammar-based representation helps mitigate such semantic issues. our main contributions are as follows: ## experiments to evaluate the performance of grammar-based code representations, we develop two sets of models, one with grammar-based code representation and one with token-based code representation. these models are built through continued pretraining from open-source code models, deepseek-coder-1.3b-base, qwen2.5-1.5b, and qwen2.5-7b, on high-quality code data. we begin by evaluating these models on code generation tasks, which are among the most widely recognized and commonly used benchmarks for assessing code-related capabilities (i.e., experiment i in 3.1). to further explore the differences between grammar-based and token-based representations, we analyze the reasons contributing to the performance gains of grammar-based representation (i.e., experiment ii in 3.2). ## conclusion in this paper, we introduce grammarcoder, a series of models trained using grammar-based code representations. to evaluate whether this approach remains effective even when billion-scale models basically no longer make syntax errors, we assess grammarcoder on widely used code generation benchmarks, humaneval(+) and mbpp(+). experimental results show that after continued pretraining on the same datasets, grammarcoder significantly outperforms models trained with normal token-based representations. to further investigate why grammar-based code representations are effective, we first quantify the differences between grammar-based and token-based approaches in representing code. additionally, we train a classification model to assess their ability to capture subtle code variations. our findings reveal that while modern llms rarely make syntax errors, grammar-based representations still enhance their ability to distinguish fine-grained token-level differences. this reduces semantic errors caused by minor variations and ultimately improves model performance in code-related tasks.