P ro C on S u L : Project Context for Code Summarization with LLM s (2024)

Abstract: We propose Project Context for Code Summarization with LLMs (ProConSuL), a new framework to provide a large language model (LLM) with precise information about the code structure from program analysis methods such as a compiler or IDE language services and use task decomposition derived from the code structure. ProConSuL builds a call graph to provide the context from callees and uses a two-phase training method (SFT + preference alignment) to train the model to use the project context. We also provide a new evaluation benchmark for C/C++ functions and a set of proxy metrics. Experimental results demonstrate that ProConSuL allows to significantly improve code summaries and reduce the number of hallucinations compared to the base model (CodeLlama-7B-instruct). We make our code and dataset available at https://github.com/TypingCat13/ProConSuL.

## introduction state of the art large language models (llms) such as @xcite , claude 3 opus @xcite , claude 3.5 sonnet @xcite , and deepseek-coder-v2 @xcite can code better than ever, exhibiting expert level capabilities in both writing code and comprehending software projects. however, they still suffer from hallucinations and may make wrong conclusions due to the lack of extended context of the entire software project. this is true for people, too: programmers need the project context to understand what a given function does, and the top-down comprehension model (understanding source code from domain context) is noisier than bottom-up comprehension (understanding code statement by statement) because it is hard for the developers to control matching current context with their domain knowledge @xcite . a straightforward solution would be to feed the entire project into the llm, but it adds a lot of unnecessary information that makes the llm's job harder and demands extra resources and special tricks to alleviate the quadratic complexity of self-attention @xcite . to reduce the amount of information needed to feed an llm, one has to find out which parts of the project context are crucial for a given practical task such as code summarization. in this work, we propose the project context for code summarization with llms framework (proconsul) that constructs precise and efficient project-level context for code summarization via formal analysis and adapts llms to this context. we focus on c/c++, a programming language very important in practice but severely underrepresented in ml research. specifically, we: (1) study different kinds of useful project context for function-level code summarization; (2) represent project context for code summarization based on code structure provided by formal analysis methods and develop a fine-tuning framework for llms with supervised fine-tuning on synthetic data and preference alignment; (3) introduce a new real life practical benchmark with automatic and semi-automatic proxy metrics for fast evaluation of code summarization quality for c/c++ and labeling instructions; (4) provide opensourced training datasets, evaluation benchmark and source code for reproducing our results 1 . the rest of the paper is organized as follows: section 2 surveys related work, section 3 introduces the method, including the proconsul framework and dataset preparation techniques, section 4 outlines the evaluation benchmarks and our experimental results, and section 5 concludes the paper. ## related work task decomposition for llms. this work was partly inspired by @xcite who intro-duced recursive book summarization, suggesting to decompose a complex task into smaller parts and then compose them back. importantly, this kind of task decomposition allowed to scale human feedback without requiring the labelers to read the whole book. in the coding domain, @xcite suggested to decompose algorithmic tasks into hierarchical natural language function descriptions and then search over combinations of possible function implementations using tests. using the formal structure of source code in llms. most previous works in this direction tried to encode code structure information into sequencebased models; e.g., @xcite uses the data flow extracted from code to help pretrain a bert-like model, while @xcite introduce a structure-induced transformer that applies regularization to the self-attention mechanism by masking the attention matrix with adjacency matrices of different graph representations. another direction of research models source code with graph neural networks (gnn) @xcite . @xcite suggest to combine self-attention layers with gnn layers, bridging the gap between global attention in transformers and inherently local gnns that rely on message passing. to extend the context to large software projects, @xcite use an agent that traverses a project's graph representation and collects information necessary to solve a specific task. in this work, we in turn use formal representations of code for task decomposition and augmenting the model context. extending the context. the problem might be solved if we were able to provide full context of a large programming project to an llm. this, however, runs into the quadratic complexity of selfattention. there are several approaches to alleviate this quadratic complexity, including sparse attention mechanisms @xcite @xcite , low-rank decomposition for the matrix of self-attention weights @xcite , or chunking attention to constrain quadratic complexity to small subsets of the input, either with a recurrent architecture @xcite @xcite , fitting more tokens by interpolating positional embeddings @xcite , or with other tricks such as hashing @xcite or blockwise attention @xcite . however, large context sizes are still challenging for llms to use efficiently @xcite , and a better solution would choose the contents of this long context wisely. evaluation metrics for text generation. most common evaluation metrics compare generated text to a reference, including n-gram-based metrics and embedding-based metrics such as rouge @xcite , meteor @xcite , or bertscore @xcite ; the problem here is the lack of high quality reference texts. one idea is to do away with them completely and evaluate based on the source document instead: @xcite thus arrive at the rouge-c metric, but in our case it is inapplicable because natural language summaries and code represent different modalities. liu et al. (2023c) and zheng et al. (2023) use a different strong llm to evaluate text generation automatically. another approach is to evaluate different properties of text separately; e.g., @xcite estimate the quality of a summary by a set of question-answer pairs automatically generated from the reference. we also note that large language models are notorious for hallucinating, i.e., introducing erroneous facts in their output, and detecting hallucinations is also an important problem @xcite . ## conclusion in this work, we have introduced the proconsul framework that gathers and uses project-level context for code summarization in c/c++, including a new method for synthetic data collection and labeling, a method for fine-tuning llms via a combination of sft and preference alignment, and a new benchmark based on real world c/c++ functions. we show that a proper use of the project context allows to significantly improve code summaries and reduce the number of hallucinations by using precise information from the context. we hope that this research is a stepping stone to bridging the gap between formal source code analysis and llms.