D oc C hecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistencies (2024)

Abstract: Comments in source code are crucial for developers to understand the purpose of the code and to use it correctly. However, keeping comments aligned with the evolving codebase poses a significant challenge. With increasing interest in automated solutions to identify and rectify discrepancies between code and its associated comments, most existing methods rely heavily on heuristic rules. This paper introduces DocChecker, a language model-based framework adept at detecting inconsistencies between code and comments and capable of generating synthetic comments. This functionality allows DocChecker to identify and rectify cases where comments do not accurately represent the code they describe.The efficacy of DocChecker is demonstrated using the Just-In-Time and CodeXGlue datasets in various scenarios. Notably, DocChecker sets a new benchmark in the Inconsistency Code-Comment Detection (ICCD) task, achieving 72.3% accuracy, and scoring 33.64 in BLEU-4 on the code summarization task. These results surpass other Large Language Models (LLMs), including GPT 3.5 and CodeLlama.DocChecker is accessible for use and evaluation. It can be found on https://github.com/FSoft-AI4Code/DocChecker and at http://4.193.50.237:5000/. For a more comprehensive understanding of its functionality, a demonstration video is available on https://youtu.be/FqnPmd531xw.

## introduction code summarization is a significant issue in software engineering due to its ability to produce explanatory comments for source code, which is essential for ensuring software quality. identifying and resolving discrepancies between the source code and its corresponding comments is an essential obstacle. inconsistencies resulting from code changes not being accurately reflected in comments or from initially imprecise descriptions can cause substantial problems in comprehending and managing software. an illustrative example of this inconsistency, sourced from the codesearchnet dataset, is depicted in figure 1 @xcite . these disparities can cause software defects, degrade software quality, and lower developer productivity, as highlighted in recent studies @xcite @xcite . moreover, the prevalent issue of code-comment conflicts in widely used datasets impacts the efficacy of code language models trained on them @xcite @xcite . recent large models trained specifically for code understanding and generation tasks may be able to address these challenges @xcite @xcite . however, these models' efficacy depends on the quality of the training data, emphasizing the importance of accurate and consistent code-comment pairs. to address these issues, we introduce doc-checker, a framework designed specifically for detecting inconsistencies between code and comments (iccd)). leveraging the capabilities of ai4se and insights gained from advancements in code llms, docchecker addresses the critical need for high-quality, consistent documentation in software development. the key idea is to leverage an encoder-decoder backbone network and then pre-train code-text pairs. this pre-training process employs a multi-faceted approach, including contrastive learning to bootstrap code and text features, binary classification to discern consistent from inconsistent pairs, and text generation to create coherent comments. the backbone of this system is unixcoder @xcite , chosen for its effectiveness and efficiency in handling multi-modal content. docchecker is specifically designed not only to detect but also to resolve inconsistencies between code and comments by generating replacement comments that accurately reflect the current state of codebase. furthermore, compared to stateof-the-art codellms, our method excels significantly on iccd and code summarization tasks. docchecker outperforms starcoder by 30% and surpasses gpt-3.5 and codellama by 10% in terms of accuracy, even though such models are pre-trained on larger-scale datasets. in summary, the key contributions of docchecker are: ## overview of docchecker in this section, we describe docchecker as a python package and demonstrate its user interface. for full customization and detailed documentation of docchecker, users can reference our github repository. ## building blocks of docchecker this section outlines the architecture of doc-checker (see section 4.1), the objectives guiding its pre-training (section 4.2), and the specific setup used during pre-training (section 4.3). initially, the model undergoes pre-training focusing on contrastive learning and code-to-text generation, followed by fine-tuning for the specific inconsistency code-comment detection (iccd) task. ## experiment setup in this section, we first present the tasks and the datasets used to assess the performance of doc-checker. then, we describe the baselines and metrics used for evaluation. ## conclusion in this paper, we present docchecker, a framework to filter and generate replacement comments for inconsistent code-comment pairs. the experimental results demonstrate the effectiveness of this method compared to sota existing methods and llms, showcasing its applicability in both academic and practical contexts. we have released docchecker as an easy-to-use library, complemented by a user-friendly interface to enhance user interaction.