H ierarchy N et: Learning to Summarize Source Code with Heterogeneous Representations (2024)

Abstract: Code representation is important to machine learning models in the code-related applications. Existing code summarization approaches primarily leverage Abstract Syntax Trees (ASTs) and sequential information from source code to generate code summaries while often overlooking the critical consideration of the interplay of dependencies among code elements and code hierarchy. However, effective summarization necessitates a holistic analysis of code snippets from three distinct aspects: lexical, syntactic, and semantic information. In this paper, we propose a novel code summarization approach utilizing Heterogeneous Code Representations (HCRs) and our specially designed HierarchyNet. HCRs adeptly capture essential code features at lexical, syntactic, and semantic levels within a hierarchical structure. HierarchyNet processes each layer of the HCR separately, employing a Heterogeneous Graph Transformer, a Tree-based CNN, and a Transformer Encoder. In addition, HierarchyNet demonstrates superior performance compared to fine-tuned pre-trained models, including CodeT5, and CodeBERT, as well as large language models that employ zero/few-shot settings, such as CodeLlama, StarCoder, and CodeGen. Implementation details can be found at https://github.com/FSoft-AI4Code/HierarchyNet.

## introduction summarizing code is crucial for aiding developers in comprehending and maintaining source code. yet, manual documentation is a laborious process. an automated method is required to craft comments efficiently. to generate precise summaries, a model should grasp lexical, syntax, and semantic * equal contribution. listing order is based on the alphabetical ordering of author surnames. † emails: minhnh46@fpt.com.vn, dqnbui.2016@smu.edu.sg, truongson.hy@indstate.edu, long.tran-thanh@warwick.ac.uk, tien.n.nguyen@utdallas.edu aspects within the code. it's imperative to capture relationships such as data and control dependencies among program elements to enhance code representation learning for code summarization. early sequence-based techniques @xcite treated code as a sequence of texts, but they did not take into account the complex interdependence of program elements in syntax or semantics. structured-based approaches @xcite @xcite were later proposed to better capture the syntactic information. the state-of-the-art approaches, such as cast @xcite and pa-former @xcite , leverage the idea of hierarchically splitting the ast into smaller parts based on its structure. cast hierarchically splits the ast's code blocks based on certain attributes, while pa-former works by treating statements as spans and splitting them into (sub)-tokens. these code-hierarchy approaches bring the benefits in terms of effective and affordable training of neural models. however, a common drawback is that they ignore the program dependencies in code representations. there are other lines of work leveraging graphs @xcite @xcite ) that model program dependencies by adding edges to the ast, in which the edges are the dependencies derived from static analysis. however, these approaches do not take into account the code hierarchy as the previous line of work. we propose a novel approach called heterogeneous code representation (hcr) to overcome these limitations by combining the strengths of both methodologies. hcr excels in encapsulating crucial code attributes across lexical, syntactic, and semantic dimensions within a hierarchical structure. this structure organizes program elements based on their features: sequences for code tokens, ast subtrees for syntax, and graphs for dependencies. significantly, we adeptly capture program dependencies by abstracting coarse-grained nodes into a higher-level layer and fine-grained nodes into a lower-level layer. this strategy enhances the generation of summaries as our model gains a more comprehensive understanding of the source code. to process our representations, we introduce a heterogeneous architecture, called hierar-chynet, which comprises a transformer encoder for processing lexical information, a tree-based encoder for processing syntactic information, and a graph-based encoder for capturing program dependencies. these layers do not operate individually but hierarchically, which intuitively captures the relationships between program elements even better. our comprehensive evaluation across diverse scenarios shows the effectiveness of our model in code summarization compared to state-of-the-art (sota) methods. our model surpasses these methods in three distinct settings: (1) hierarchical neural networks (nns) of code akin to us, such as paformer and cast; (2) fine-tuned sota pretrained language models of code, such as codet5 and codebert; and (3) in-context learning of large code language models using zero-shot and fewshot settings, such as starcoder and codegen. to summarize, our key contributions include: ## related work code summarization research in generating the descriptions for source code has evolved through various techniques. initially, sequencebased methods treated code as text @xcite @xcite , disregarding syntactic or semantic dependencies among program elements. for example, neu-ralcodesum @xcite ) is a purely transformer-based approach that receives code tokens and generates summaries. structure-based and tree-based approaches were also proposed to capture the syntax of source code @xcite @xcite @xcite @xcite . for instance, treel-stm @xcite employs bottom-up node accumulation, while tptrans @xcite integrates ast path information into the transformer. cast @xcite and pa-former @xcite are currently the state-of-the-art methods with the same key idea of breaking the code into a structural hierarchy. finally, graph-based techniques were used to capture code semantics by adding inductive bias into the ast through semantic edges, turning it into a graph @xcite @xcite . however, they still encounter challenges in representing code hierarchy and program dependencies, as well as neural networks to handle them. pretrained language models for source code besides code summarization, language models of code generally support various code understanding tasks, such as code generation @xcite @xcite , code completion @xcite @xcite , etc. a large body of recent work employs language models from natural language processing for code @xcite @xcite @xcite @xcite @xcite @xcite , applying similar pretraining strategies as used for natural languages. despite their promising performance, these pretrained models have not been empirically demonstrated to effectively capture semantics in source code, such as data, control flows, and other program dependencies among code elements. in contrast, incorporating code-specific features into representations as inductive biases has been shown to increase the model's knowledge @xcite . importantly, the control and data dependencies among the statements could provide valuable insights into the intended execution order. ignoring control dependencies hinders the model's ability to capture such intentions because the sequential order in the code may not reflect the execution order. for example, despite their sequential order, the execution of the statement at line 6 is not guaranteed to follow the statement at line 5, as it is dependent on the outcome of the if condition at line 5. moreover, the data dependency via the variable num at line 5 and line 6 is also crucial for summarization as it indicates that only positive numbers are collected. previous approaches, such as those outlined in leclair et al. (2020), fernandes et al. (2019), and hellendoorn et al. (2020a) , utilize heuristics from static analysis to connect nodes in the ast to represent dependencies. however, the large size of the ast can pose challenges for a model to effectively capture dependencies among distant nodes @xcite . in contrast, state-of-the-art approaches such as cast @xcite and pa-former @xcite create a hierarchy among code elements by splitting the ast into smaller parts. however, these methods do not maintain program dependencies among the elements. we propose the heterogeneous code representation (figure 1b ) to restructure code into hierarchical layers, abstracting meaningful entities such as statements or expressions into single nodes in a higher layer. importantly, hcr also enables the representation of dependencies, including data, control, sequential, and syntactic dependencies. we introduce a heterogeneous neural network utilizing an appropriate neural network at each level: a transformer encoder for code tokens, a tree-based encoder for ast subtrees, and a graph neural network for the coarse-grained dependencies. this approach reduces the computational workload and improves capturing the dependencies between distant nodes in an ast (an issue with the prior works). ## heterogeneous code representation this section presents the heterogeneous code representation (hcr) that integrates both hierarchical structure of source code as well as program dependencies among program elements. figure 2 (left-side) displays the three layers of heterogeneous code representation (hcr). the first layer, denoted by the "linearized ast sequence," is a sequence of nodes l from the serialization process of the abstract syntax tree (ast) of the given program. the second layer, the "subtree-level," represents the statement and expression-level program elements, each represented by a significantly smaller subtree t ## neural network architecture this section explains the neural network architecture for our hierarchynet method (figure 2 ). each node l i in a sequence of nodes l has two attributes: token and type. the initial representation of each node l i is computed by concatenating the embeddings of its token and its type. these embeddings can be looked up from two learnable embedding matrices (token and type). we denote s i be the initial embedding of the node l i , @xmath0, 0 < @xmath1. the neural network architecture, hierar-chynet, consists of the following components. ## empirical evaluation we have conducted several experiments to evaluate hierarchynet. we seek to answer the following research questions: datasets. to ensure a comprehensive comparison with several sota baselines, we considered multiple well-established datasets for code summarization, namely tl-codesum @xcite , deepcom @xcite , funcom @xcite , and funcom-50 @xcite . note that different baselines use distinct datasets and achieve sota results. the funcom-50 dataset was used by pa-former @xcite , but with a number of samples filtered out from fun-com, approximately 50% of the data. we followed the original dataset's partition in funcom @xcite for training, testing, and validation. baselines. we compared hierarchynet against three categories of baselines. the first category includes the baselines trained from scratch without utilizing pretrained checkpoints. examples include cast @xcite and pa-former @xcite , which are consciously designed to incorporate code structures. additional baselines in this category, grouped by code representation and neural architecture, including sequence-based models (ncs @xcite ), structure-based and tree-based models (astattgru @xcite , hdeepcom @xcite , tptrans @xcite , codeastnn @xcite , sit @xcite ), and graph-based models (great (hellendoorn et al., 2020a)). the second category comprises fine-tuned baselines for code summarization from well-known pretrained models. for representative models, we fine-tune codet5-base @xcite and codebert-base @xcite , considering codet5 as the state-of-the-art for code summarization and codebert as a widely-used model. the third category encompasses large language models that can perform in-context learning for code understanding tasks using zero-shot, one-shot, or few-shot learning approaches. for this category, we used codellama 13b @xcite , starcoder @xcite and codegen-multi 2b @xcite . metrics. we employ bleu @xcite , meteor @xcite , rouge-l @xcite , cider @xcite and f1-score, which are commonly used as the evaluation metrics for code summarization. results. the results shown in in conclusion, the results show that hierar-chynet, which utilizes a hierarchical-based architecture and dependencies information, significantly improves code summarization performance. ## conclusion we introduce an innovative framework for code summarization that combines heterogeneous code representations (hcrs) with hierarchynet, a neural architecture tailored for processing hcrs. our hcrs capture critical code attributes across lexical, syntactic, and semantic levels by organizing coarse-grained code elements into a higherlevel layer while integrating fine-grained program elements into a lower-level layer. hierarchynet is engineered to handle each layer of the hcr independently, enabling the representation of information gathered at the fine-grained level as input at the coarse-grained level. the core concept of hierar-chynet lies in integrating multi-level code representations and program dependencies. our empirical evaluations demonstrate that our approach surpasses various state-of-the-art techniques across diverse settings, including structure-based models (cast, pa-former), fine-tuned pretrained models (codet5, codebert), and in-context learning (codellama, starcoder, codegen). our ablation study shows that all of the components in hier-archynet contribute positively to its high performance. we also conducted a human study to evaluate the code summarization results produced by hierarchynet. the results show that human subjects highly regarded the code summarizaiton results from our model.