C ode E xp: Explanatory Code Document Generation (2022)

Abstract: Developing models that can automatically generate detailed code explanation can greatly benefit software maintenance and programming education. However, existing code-to-text generation models often produce only high-level summaries of code that do not capture implementation-level choices essential for these scenarios. To fill in this gap, we propose the code explanation generation task. We first conducted a human study to identify the criteria for high-quality explanatory docstring for code. Based on that, we collected and refined a large-scale code docstring corpus and formulated automatic evaluation metrics that best match human assessments. Finally, we present a multi-stage fine-tuning strategy and baseline models for the task. Our experiments show that (1) our refined training dataset lets models achieve better performance in the explanation generation tasks compared to larger-scale unrefined data (15x larger), and (2) fine-tuned models can generate well-structured long docstrings comparable to human-written ones. We envision our training dataset, human-evaluation protocol, recommended metrics, and fine-tuning strategy can boost future code explanation research. The code and annotated data are available at https://github.com/subercui/CodeExp.

## introduction code documentation improves program comprehension @xcite and reduces software maintenance cost @xcite . recently, many automated code summary tools have been developed to reduce the effort of document creation: for example, denigma 1 is an ide extension for generating inline function summaries; github copilot labs 2 is a code summary model built on top of codex @xcite for generating explanations for ai generated code. however, these existing code summary tools focus on the generation of short high-level descriptions of source code semantics @xcite @xcite , and code summary alone is insufficient to meet the software understanding and maintenance need: a recent survey shows 85% developers expect tools to generate method-level documentations explaining the functionalities, usage and design rationales of code @xcite . for example, as shown in figure 1 , the code summary captures only the high-level code functionality, while the explanatory docstring explains arguments, return values, and its computation process in a detailed and informative way. however, due to the lack of training and evaluation resources, few code explanation models have been developed. in this work, we introduce the code explanation generation task. we provide (1) the training corpus, (2) fine-tuning strategy, and (3) humanevaluation protocol and recommended automatic evaluation metrics to support developing code explanation models. our contributions include: returns a random number generator. the rng object is generated using the first of these cases that produces a valid result: 1) rng_or_seed itself 2) constructor(rng_or_seed) 3) constructor(default_seed) 4) constructor(42) parameters: rng_or_seed (int or rng): if `rng_or_seed` is a random number generator, then it is returned. if `rng_or_seed` is an integer, then a random number generator is created using `constructor` and seeded with `rng_or_seed`. default_seed (int): seed used if rng_or_seed is none. constructor (function or class): must return a rng object. constructor is called with rng_or_seed, default_seed or 42. returns: an rng object. formance and highlights the importance of highquality data for the code explanation task. ## code explanation generation application scenarios. we focus on the generation of code explanations that describe both lowlevel and high-level code semantics. the automatic code explanation tool can benefit developers in many scenarios. for example, the tool can reduce software engineers developing effort by automatically generating function comments during development; it can help learners and codebase maintainers better understand undocumented code; it can also explain code generated by code generation models like codex for the developer to better understand and verify its correctness. note that in these scenarios, because the developer needs to understand both design rationale and implementation details of the code, an explanation covering detailed code semantics would be more appropriate than a short high-level description. for example, if a developer aims to create a test for the function make_rng in figure 1 when maintaining a codebase, it is crucial to understand how the variable rng_or_seed is defined and used. ## codeexp data collection this section describes our data selection process. the code explanation corpus (codeexp) consist of three sets of code-docstring pairs: (1) code-exp(annotated) with 13k human annotated pairs. (2) codeexp(refined) with 158k pairs filtered by a trained model. (3) codeexp(raw) with 2.3m unlabelled pairs. ## experiment settings in this paper, we formulate the code explanation generation as a sequence-to-sequence problem, where the source sequence is a function code (including both function signature and body), and the target sequence is an explanatory documentation string. we select four strong pretrained language models for programming language and fine-tune the models on our proposed codeexp dataset. we next introduce the baseline models, experiment settings, and evaluation metrics. ## metric selection based on human evaluation we conducted human evaluations on generated docstrings and compared the aforementioned automatic metrics against the human-eval results. the protocol of our human evaluation is designed to cover four important aspects: a1. general adequacy, a2. coverage, a3. coherence, a4. fluency. annotators rate for each aspect a score within 0-4 on the 5-point likert scale @xcite . the detailed setup is listed in appendix a.4. aspects 1,3,4 have been used in both machine translation @xcite and code summarization studies @xcite . the coverage aspect emphasizes the preference for informative explanations of code pieces. notably, these scores provide reference-free assessments. both the original reference docstring (identity is hidden) and generated ones are provided to annotators. we calculate the adapted kendal's τ @xcite to measure the agreement between automatic metrics and human evaluation. for an arbitrary pair of two examples, it considers whether two metrics both prefer one example to the other. the τ value is the ratio difference of concordant (con) and discordant (dis) pairs, ## related works several approaches have been proposed for method/function-level automatic code documentation. early studies use template-based and information retrieval approaches to generate longform documents @xcite @xcite . recent efforts have been mainly focused on the summarization. @xcite collected a large parallel python code and docstring corpus and trained lstm-based machine translation model. deepcom @xcite introduced structure information of ast to help generate summaries for java methods. @xcite proposed contectcc, which encodes the context information of external dependencies using the api calls in the source java method. more recently, ahmad et al. (2020) proposed a transformer approach for method level summarization. @xcite built a retrieval-based approach using similar code snippets in the generation. the aforementioned studies employed several popular pl-nl corpora @xcite @xcite . the average length of these corpora is below 20 tokens and they target the summarization task where the docstrings are often oneliners. apart from these efforts, few deep learning based approaches generate full-length of documentation. @xcite pretrained t5 models to generate python docstring in numerous styles. the openai codex @xcite trained gpt-3 for both code and document generation in six programming languages. in these mentioned studies, machine translation (mt) metrics have been widely used for code comment assessments. @xcite questioned this adoption of reference-based mt metrics by examining and showing the semantic difference between code and nl. @xcite rigorously examined the consistency between automatic metrics and human assessments using kendall's τ as well and recommended the usage of bleu, me-teor, and chrf. ## conclusion the code explanation generation is an important task for code understanding. on one hand, we show existing summarization methods do not directly apply to this task. on the other hand, we built data collection pipelines, explored consistency between automatic and human evaluations, and provided a framework for fine-tuning existing pretrained models to generate explanatory docstrings of human-comparable quality. we highlight the importance of data quality by showing that fine-tuning on high-quality data exceeds the performance using raw data of 15 times larger scale. we expect the proposed infrastructures, including the annotated dataset, human-evaluation protocol, recommended metrics, and fine-tuning strategy, to boost future research for code explanation.