A Neural Network Architecture for Program Understanding Inspired by Human Behaviors (2022)

Abstract: Program understanding is a fundamental task in program language processing. Despite the success, existing works fail to take human behaviors as reference in understanding programs. In this paper, we consider human behaviors and propose the PGNN-EK model that consists of two main components. On the one hand, inspired by the “divide-and-conquer” reading behaviors of humans, we present a partitioning-based graph neural network model PGNN on the upgraded AST of codes. On the other hand, to characterize human behaviors of resorting to other resources to help code comprehension, we transform raw codes with external knowledge and apply pre-training techniques for information extraction. Finally, we combine the two embeddings generated from the two components to output code embeddings. We conduct extensive experiments to show the superior performance of PGNN-EK on the code summarization and code clone detection tasks. In particular, to show the generalization ability of our model, we release a new dataset that is more challenging for code clone detection and could advance the development of the community. Our codes and data are publicly available at https://github.com/RecklessRonan/PGNN-EK .

## introduction the past decades have witnessed the prosperity of programming platforms, such as github and stack overflow. these platforms generate massive open-source code 1 data that is named as "big code" in @xcite . to automate the software development and maintenance, based on the "software naturalness" hypothesis @xcite , natural language processing (nlp) techniques have been applied in program understanding. after that, a series of downstream programming language processing (plp) tasks can be performed, including code summarization @xcite @xcite and code clone detection @xcite . existing works for understanding programs mainly utilize three types of information: code context, code structure and external knowledge. specifically, code context refers to the token sequence in the code. for code structure, each code can be parsed into various types of intermediate representations, such as ast (abstract syntax tree), cfg (control flow graph) and pdg (program dependence graph). these representations capture the structural information of codes. further, there also exists external knowledge associated with codes, such as api documentation and other exemplary codes. despite the success, all these models ignore considering human behaviors in reading programs. recently, @xcite suggest the potential futures of deep learning by comparing current ai methods with human learning abilities. this further prompts us to revisit program understanding: can we develop a model that understands programs like humans? in the domain of programming education, how people understand codes is a topic that has been studied. for example, based on knowledge base including syntactical knowledge (e.g., programming basics) and semantic knowledge (e.g., api documentation), @xcite offer a bottom-up reading technique, which assumes that people begin with individual code lines and chunks, and then combine them into higher-level abstractions. further, @xcite state that when people read codes, reasoning about the hierarchical relationship of blocks, statements, expressions and variables is necessary. based on these studies, we conclude three key points for human understanding codes. first, the transition of defined variables has to be traced. second, humans usually adopt a "divideand-conquer" strategy, which divides codes based on statements and then understands codes from a local-to-global view. third, humans resort to external knowledge to comprehend codes, such as api documentation and code examples written by experts. in this paper, inspired by human behaviors for code comprehension, we propose a novel partitioning-based graph neural network with external knowledge (pgnn-ek). to capture code context and structure, pgnn-ek upgrades the traditional ast and defines a novel subtoken-based ast called s-ast. in s-ast, we add edges between variables to trace the variable transitions, edges between adjacent tree leaves from left to right to enrich the context and structure information, and edges between sub-nodes corresponding to subtokens tokenized from user-defined identifiers to handle the out of vocabulary (oov) problem @xcite ). details will be illustrated later. after that, we first apply graph neural network (gnn) models on the s-ast to derive a code embedding. to further implement the "divide-and-conquer" reading strategy, we partition the s-ast into multiple subgraphs, which follow the sequence of statements in the original code. for each subgraph, we use gnn models to generate the subgraph embedding. then, these subgraph embeddings are fused to generate another code embedding. for these two code embeddings, since they are both derived from s-ast, we further aggregate them. on the other hand, to characterize the dependence on external knowledge for code comprehension, we traverse the ast of the original code to derive a sequence of tokens for syntactic knowledge and then add the api descriptions to the end for semantic knowledge. we then apply code-bert @xcite on the token sequence to capture external knowledge. finally, pgnn-ek generates the output code embedding by combining the embedding derived from s-ast and the one from external knowledge. to evaluate the model performance, we conduct experiments on the code summarization task and code clone detection task, respectively. before we apply pgnn-ek on the code clone detection benchmarks in codexglue @xcite extracted from the bigclonebench 2014 dataset @xcite , we notice from the leaderboard foot_0 that the results are incredibly high, where the minimum f1 score is 0.949. then we dive into the characteristics of the dataset and find that the functionalities of codes in the test set have all appeared in the training set. therefore, the dataset is very simple. to further test the model's generalization ability, we construct a new dataset, where the test set contains codes whose functionality has never appeared in the training set. this new dataset provides an insightful reference for further research in the community. our main contributions are summarized as follows: in particular, we identify the limitation of a benchmark dataset for code clone detection and release a new dataset that is more challenging. ## s-ast construction in this section, we construct s-ast. the original ast has two main limitations: after we split the nodes by camel case and underscores @xcite , the vocabulary size is still as high as 201, 286. a very large vocabulary could cause the oov problem @xcite and thus adversely affect the model performance. to improve the connectivity of the ast, there exist some works @xcite @xcite that add edges to the ast. however, these methods cannot address the oov problem. therefore, we propose a new code intermediate graph s-ast, as shown in figure 1 . similar as in @xcite , we add data flow edges to trace variable transitions and connect adjacent leaf nodes to encourage learning from contexts. to solve the oov problem, we further reduce the vocabulary size by using the tokenizer of roberta @xcite to tokenize every leaf node in the ast. when a leaf node can be tokenized into multiple subtokens, we keep the first subtoken as the parent node and take other subtokens as its children. for example, the token "getlarger" is divided into the parent node "get" and the children nodes "l" and "arger". these new parent-children connections are defined as subtoken edges. with these three types of edges added, we increase the number of edges in the ast and improve the graph connectivity. further, the vocabulary size could be significantly reduced. in our experiments, we use javalang foot_1 to generate java ast and reduce the vocabulary size to 50, 336, where 50, 265 is the size of original roberta vocabulary and 71 is the number of keywords in non-leaf nodes defined by javalang. ## algorithm in this section, we introduce the pgnn-ek model, which is composed of two main components. on the one hand, the partitioning-based graph neural network model (pgnn) is proposed to follow the "divide-and-conquer" behaviours of humans to understand programs. on the other hand, pgnn-ek leverages external knowledge to enhance the model's capability. the overall architecture of pgnn-ek is summarized in figure 2 . ## experiments in this section, we evaluate the performance of pgnn-ek. we conduct experiments on two program understanding tasks: code summarization and code clone detection. for each task, we use two benchmark datasets, whose statistics are listed in ## conclusion in this paper, we followed human understandings for programs and proposed the pgnn-ek model. to enrich the code structure information and alleviate the oov problem, we presented the s-ast graph based on ast, which uses a subtoken tokenizer to generate subtoken nodes and edges between them. inspired by the "divide-and-conquer" strategy, we proposed the partitioning-based graph neural network model on s-ast that employs code context and structure. to leverage the external knowledge to boost comprehension, we transformed the raw code to fuse syntactic and semantic knowledge and utilized pre-training techniques for information extraction. we performed extensive experiments to show the effectiveness of our model pgnn-ek on the code summarization and code clone detection tasks. in particular, to show the generalization ability of the model, we released a new benchmark that is more challenging.