Benchmarking Language Models for Code Syntax Understanding (2022)

Abstract: Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding, which represent the input as a token sequence without explicitly modeling its structure. Some prior works show that pre-trained language models can capture the syntactic rules of natural languages without finetuning on syntax understanding tasks. However, there is limited understanding of how well pre-trained models understand the code structure so far. In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs. Specifically, we introduce CodeSyntax, a large-scale dataset of programs annotated with the syntactic relationships in their corresponding abstract syntax trees. Our key observation is that pre-training on massive code data does not result in decent code syntax understanding. In fact, these pre-trained programming language models fail to match the performance of naive baselines based on positional offsets and keywords. We also present a natural language benchmark to highlight the differences between natural languages and programming languages in terms of understanding corresponding syntactic structures. Our findings point out key limitations of existing pre-training methods and suggest the importance of modeling syntactic structures for the programming language.

## introduction large-scale pre-training of language models has become the de-facto paradigm for a variety of natural language processing tasks. furthermore, recent studies show that models pre-trained on a massive amount of code also achieve competitive performance on many tasks, e.g., code generation and † corresponding authors. 1 our code and dataset are available at https://github. com/dashends/codesyntax . there were many pioneer pc contributors. offset bert roberta codebert english syntax understanding 47.1 64.6 62.7 59.3 offset cubert codebert roberta python syntax understanding 43.6 39.2 33.1 34.5 figure 2: a preview of the model performance comparison on nl and pl syntax understanding tasks. pretrained models capture nl syntax relatively well, but perform worse in understanding pl syntax. the offset baseline picks the token using a fixed positional offset. we use bert-large and roberta-base configurations (corresponding to the configurations of cubert and codebert). the plot shows top-1 scores. see tables 3 and 4 for the full results. in this work, we investigate whether large-scale pre-training is all we need for code representation learning. in particular, we conduct the first systematic study to analyze how the pre-trained language models understand the syntactic structures of programs. to this end, we introduce codesyntax, a large-scale benchmark consisting of programs annotated with the syntactic relationships between different tokens. the ground truth syntactic relationships are extracted from edges in the abstract syntax trees (ast) of the programs. figure 1 shows some examples. these syntactic relations are function-wise similar to dependency relations for nl, where prior work has demonstrated that the attention heads of pre-trained language models can help to identify nl relation types @xcite . to measure how well the pre-trained language models capture the code syntactic structures, we adopt the approach to the pl domain. we focus on investigating the zeroshot capability of existing pre-training methods in our experiments, and we evaluate these pre-trained models without finetuning them on our benchmark. we evaluate the state-of-the-art pre-trained language models for code representation learning, including cubert @xcite and code-bert @xcite . a common characteristic of these models is that they share the same transformer-based architectural design as nl models @xcite . this allows us to directly compare their performance in capturing the syntax structure. we present a preview of our key results in figure 2 . our main observation is that pre-training is insufficient for learning the syntactic relations in code. first, we find that the models pre-trained on code do not always outperform models pre-trained on nl corpus alone. surprisingly, compared to codebert which is trained on both text and code corpora, roberta achieves better performance without training on any code with identical model architecture. this indicates that pre-training on programs as token sequences does not help learn the syntactic relations. on the contrary, without dependency rela-tions, pre-training still enables language models to understand the nl syntax to some extent. moreover, for code syntax understanding, the pre-trained models even perform worse than simple baselines that pick the tokens with a fixed offset. for example, always selecting the (p+2)-th token as the p-th token's dependency yields higher accuracy than any attention head for several relation types. on the other hand, the same model architectures pre-trained on text corpora achieve decent accuracy in identifying the dependency relations in the nl domain, where the performance of the same simple baselines is far behind. our analysis reveals several key differences between nl and pl that lead to different capabilities of understanding the syntax for pre-trained models. first, programs are more structured than nl sentences. programs usually contain hierarchical structures representing long-term dependencies between code tokens. consequently, a large number of syntactic relation types are between distant tokens, which can be difficult to recognize for attention heads. on the contrary, the dependency relations in nl sentences mostly connect nearby token pairs, and in this case the attention heads are more capable of identifying the correct relations. meanwhile, language models are good at recognizing keyword-based relations, such as picking the corresponding else keyword for an if token. interestingly, we find that the inclusion of tokens such as newlines and semicolons notably affects the performance in the code domain. our findings suggest that existing pre-trained models perform quite differently in pl and nl domains in terms of the ability to understand syntax. thus, directly applying training paradigms developed for nl could be suboptimal for program learning, and we consider designing better approaches to model the code structure as future work. ## evaluation setup do pre-trained language models capture the code structure without direct supervision of the syntactic information? to investigate this question, we evaluate several pre-trained language models without finetuning, and compare their performance in understanding the syntax for nl and pl. natural language benchmark. to compare the performance on codesyntax to nl syntax understanding, we construct the nl benchmark that includes english and german. specifically, we use the english news text treebank: penn treebank revised @xcite labeled with stanford dependencies (de marneffe and manning, 2008a,b), and german hamburg dependency treebank (foth et al., 2014) labeled with universal dependencies (de marneffe et al., 2021). in total, the english dataset has 48,883 sentences, 43 relation types, and 1,147,526 relation edges; the german dataset has 18,459 sentences, 35 relation types, and 307,791 relation edges. attention probing approach. some prior works demonstrate that a transformer architecture @xcite pre-trained on a text corpus, such as bert @xcite , contains attention heads that specialize in certain dependency relations in nl @xcite . specifically, in the transformer architecture, each vector e i for an input token is transformed into the query and key vectors q i and k i via some linear transformations, and the transformations vary among different attention heads. for the i-th token, the attention weight assigned to the j-th token is the attention weight indicates how important the j-th token is with respect to the i-th token. typically, different attention heads learn different weights between input tokens. therefore, to measure the correctness of recognizing a relation type r, for each @xmath0, t, @xmath1, we enumerate all attention heads to compute the attention weight α h,t . if an attention head tends to assign high attention weights that connect the pair of tokens belonging to the relation type r, we consider the relation type to be captured. we defer more implementation details of attention map extraction to appendix b. metrics. we use the unlabeled attachment score (uas) to measure the syntax understanding performance, and we consider top-k scores with different values of k. to compute top-k scores for language models, for each attention head, given the head token h in a relation @xmath2, t, r>, we compute the attention weight over all tokens in the input code, and we consider the prediction to be correct if the attention weight over the dependent token t is among the top-k tokens with the highest attention weights. for each relation, we select the best-performing attention head and use its score as the model's score for that relation. we calculate a model's average score over all relations as the final score of the model. in nl dependency parsing problems, the dependent node t usually corresponds to a single word. however, in pl, the dependent can be a block that contains multiple code tokens. for example, in the if:@xmath3, the head is the keyword if, while the dependent is the entire body block. therefore, we measure three metrics. first-token metric and last-token metric: the prediction is deemed correct if it successfully predicts the first and last token of the dependent block, respectively; anytoken metric: the prediction is considered correct if it can predict any token within the dependent block. while we agree that these are not perfect metrics and one single metric may be incomplete, we observe that our findings generally hold for all the three metrics we evaluated. note that the firsttoken metric is stricter than the any-token metric by design. unless otherwise specified, we report the top-k scores using the first-token metric by default. model architectures. baselines. to examine how well the attention performs through comparisons, we design a simple offset baseline and a simple keyword baseline. the offset baseline with an offset value of i always selects the token after i positions of the input token as its prediction when @xmath4, and selects i positions before the input token when @xmath5. the keyword baseline with a keyword of key always predicts the next key token as its prediction. in our experiments, we evaluate offset baselines with each possible offset value between 0 and 512 for pl, and -512 to 512 for nl. we use all python and java keywords for the keyword baselines on python and java datasets respectively, including tokens such as if, for, in, etc. to evaluate the top-k scores for baselines where @xmath6, we combine k simple baselines with different offset (keyword) values to give k predictions. to select k offset (keyword) values, we repeatedly and greedily include the next value that yields the highest performance increase for the relation type under consideration. ## experiments in this section, we present the results of pre-trained language models for both pl and nl syntax understanding tasks, and discuss the key observations that distinguish pl from nl. ## related work transformer-based language models have been widely used for natural language processing @xcite @xcite @xcite . @xcite show that syntax trees are implicitly embedded in bert's word representation space via a structural probe. another line of work studies what is learned by the attention in language models @xcite @xcite @xcite @xcite . in particular, @xcite evaluate the attention heads of bert on dependency parsing tasks using the english penn treebank corpus, where the attention significantly outperforms offset baselines. on the contrary, we demonstrate that attention-based models largely perform worse than offset baselines on code syntax understanding. the success of transformer-based models for natural language processing leads to their application in the pl domain @xcite @xcite @xcite . @xcite evaluate the model performance by measuring the functional correctness on unit tests. @xcite empirically shows that transformers can utilize syntactic information to make predictions in some code processing tasks, while we analyze attention's ability to understand syntactic relations. @xcite probe pre-trained models on four code understanding tasks. they focus more on code classification, e.g., they train a classifier for predicting the ast node tag and the code length. on the contrary, we probe the attention heads for syntactic relation understanding, and we aim to present a comprehensive study of the differences between pre-trained language models on nl and pl for capturing the syntax structures. there have been some efforts that try to take code structure into account during pre-training of transformer-based models for code. for example, @xcite utilizes data flow for pretraining; i.e., the relation of "where-the-value-comes-from" for variables. on our python benchmark, graphcodebert achieves a top-1 first-token score of 39.3, which is better than 33.1 of codebert, and comparable to 39.2 of cubert. however, such a score is still worse than 43.6 of the offset baseline. this trend is consistent when evaluating with other metrics. these results show that pre-training on data flow helps improve the model's ability to understand code syntax, but there is still large room for improvement. ## conclusion in this work, we introduce codesyntax, a largescale benchmark for measuring the performance of code syntax understanding. based on codesyntax, we conduct the first comprehensive study to analyze the capability of pre-trained language models on understanding the code syntactic structures without further finetuning. we demonstrate that while the attention heads of pre-trained language models are able to identify dependency relations in nl to some extent, they have difficulty recognizing the syntactic relations in programs. pre-trained models even generally perform worse than simple offset baselines, and they tend to attend to frequently occurring nearby tokens without taking the hierarchical code structure into consideration. we also analyze the differences between nl and pl from the perspectives of pre-trained models. our evaluation suggests that pl has unique characteristics that distinguish them from nl, such as the long-term dependency between code tokens, and the hierarchy in the syntactic structures. therefore, simply taking a program as a token sequence is insufficient for modeling the program structure, which could eventually limit the potential of language models for code understanding tasks. we consider developing new model architectures and pre-training algorithms to leverage and represent the code structure and dependency graph as important future work.