C ode QA : A Question Answering Dataset for Source Code Comprehension (2021)

Abstract: We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.

## introduction question answering (qa) is the task of answering questions given a context about which the questions are being asked. with the advancement of deep learning and the availability of large-scale data, qa has received increasing attention from researchers. in recent years, qa has been applied into broad application domains, such as news @xcite , science @xcite , movies @xcite , medical field @xcite , etc. among qa's wide applications, code qa is an appealing application scenario on account of the distinctive nature of code differing from text. in this study, we focus on generating qa pairs for source code for the purpose of source code comprehension. qa-based source code comprehension is the ability to read a code snippet and then answer questions about it, which requires understanding both source code and natural language. take the public void insertchildat(element child, int index){ setchildparent(child); children.add(index, child); } question: what does the code insert at the specified index? answer: the given child. compared with code summarization task @xcite that generates comments for codes, qa-based code comprehension task introduces more specific guidance and more explicit signals for models on what to generate. it provides more granularity levels ranging from method to variable, not just regarding several lines of code as a whole. besides, it is easier to be evaluated since the output is more succinct, constrained and targeted @xcite . qa-based source code comprehension has direct use in education to facilitate programming learning, where a system automatically answers questions about codes that someone has read. a more general use is to help improve software maintenance since it can advance the readability of code. moreover, it can provide diverse information that can be leveraged to help perform a wide range of software engineering tasks, such as bug detection, specification inference, testing and code synthesis. however, constructing a code qa dataset for source code comprehension is very challenging. naturally occurring qa pairs on the web are often complicated, noisy and contain information that cannot be inferred from the source code. we tried to collect naturally occurring qa pairs from source code management platforms like github, qa sites for programmers like stackoverflow, programming online judge platforms like leetcode. but these qa pairs usually rely on knowledge apart from source code, which makes it problematic to disentangle modeling weakness from data noise. an alternative is to have experienced developers write qa pairs for source code from scratch, which is inefficient and cost-intensive. in this study, we introduce a new data construction process to address the above challenges and propose codeqa, a free-form question-answering dataset. first, to ensure that qa pairs are natural and clean, we utilize code comments as data source. we pick out two large-scale well-studied datasets from github -a java dataset and a python dataset. then we select comments that are suitable to generate qa pairs from these datasets. targeting at generating various types of qa pairs such as wh-questions and yes/no questions, we implement syntactic rules and semantic analysis to transform comments into qa pairs. more specifically, comments are transformed into dependency trees and converted to fit question templates that are invoked by semantic role labels. we also analyze the verbal group of comments and generate yes/no questions. after that, qa pairs with ambiguous answers and unbalanced counts of yes/no are filtered. due to the varied nature of code comments, cod-eqa covers a variety of information containing in codes, ranging from method to variable. we analyze our dataset and classify all the generated qa pairs into four categories: functionality, purpose, property and workflow. our experiments with several baseline models demonstrate that neural models struggle to generate correct answers. these results suggest that our dataset could serve as a useful groundwork for qa-based source code comprehension. prior work @xcite built a dataset for code qa, but only a third of their questions are free-form. the main differences between our work and prior work are: first, all of our questions are free-form; second, our questions have diverse textual expressions and they are asking about information of various granularity in code, while questions in prior work are mostly fixed and on the single granularity. therefore, the contributions of this paper are as follows. experimental results demonstrate this dataset could serve as a useful benchmark for model and metric development. ## the code qa task in this work, we focus on building a dataset and setting up baselines for the following code qa task: given a source code c and a natural language question q about c, a free-form textual answer a is required to be generated. the textual answer a may be a word, a phrase or a sentence, and it usually cannot be directly extracted from the source code. this task is different from traditional machine reading comprehension, as code is very different from text and we need programming knowledge to understand it. the source code and the natural language question are actually in two different languages and a qa system should have the ability to understand both the code language and the natural language. moreover, the system needs to generate an answer faithful to the question and the corresponding code, rather than extract some tokens from the code. therefore, the task is very challenging and it is very meaningful and urgent to construct and release a large-scale dataset for this task. ## dataset analysis in this section, we introduce the overall statistics of our dataset and verify the free-form characteristic of our dataset. to explore the distribution of different kinds of source code qa pairs, we propose a taxonomy of four types of source code comprehension. ## experiments in this section, we first introduce four baseline models for this task. then we compare and analyze the results of different models under both automatic and human evaluation metrics. ## conclusion in this paper, we build the first diverse free-form question answering dataset for code by transforming code comments into qa pairs. we also provide several neural baselines, and demonstrate that cod-eqa could lay the foundation for further research on qa-based source code comprehension. for future work, how to expand the number of question types and generate more high-quality qa pairs are the major challenges. besides, we will explore more powerful qa models to better leverage information from code and capture interaction between code and question.