CS 1 QA : A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course (2022)

Abstract: We introduce CS1QA, a dataset for code-based question answering in the programming education domain. CS1QA consists of 9,237 question-answer pairs gathered from chat logs in an introductory programming class using Python, and 17,698 unannotated chat data with code. Each question is accompanied with the student’s code, and the portion of the code relevant to answering the question. We carefully design the annotation process to construct CS1QA, and analyze the collected dataset in detail. The tasks for CS1QA are to predict the question type, the relevant code snippet given the question and the code and retrieving an answer from the annotated corpus. Results for the experiments on several baseline models are reported and thoroughly analyzed. The tasks for CS1QA challenge models to understand both the code and natural language. This unique dataset can be used as a benchmark for source code comprehension and question answering in the educational setting.

## introduction question answering (qa) studies systems that understand questions and the relevant context to provide answers. question forms include single document qa @xcite , multi-hop qa @xcite , conversational qa @xcite , and open domain qa @xcite . questions about specific domains are asked in newsqa @xcite and techqa @xcite , and images are provided with the question in visual qa @xcite . another interesting field of qa asks questions about source code @xcite . a useful application of qa is in the educational domain. asking questions and getting the answer is an essential and efficient means of learning. in this paper, we focus on qa for programming education, where both the input modes and the domain pose interesting challenges. answering these questions requires reading and understanding both source code and natural language questions. in addition, students' questions are often complex, demanding thorough understanding of the context such as the intention and the educational goal to answer them. recently, models that understand programming languages (pl) have been studied, and show promising results in diverse code comprehension tasks @xcite @xcite . however, these models have limitations to support question answering. they are not trained on datasets containing questions about the code and are not designed for qa tasks. also, many assume fully functional code as input, while students' code contains diverse syntax and logical errors and is often incomplete. to address this issue, we introduce cs1qa, a new dataset with tasks for code-based question answering in programming education. questions and answers about programming are collected from the naturally occurring chat messages between students and tas. the question type and the code snippet relevant to answering the question are also collected. the final cs1qa dataset consists of question, question type, answer, and code annotated with relevant lines. the data is collected mostly in korean and then machine-translated into english and quality-checked for easy application on models pretrained in english. figure 1 shows an example of our data. we also include two-semesters' worth of ta-student chat log data consisting of 17,698 chat sessions and the corresponding code. we design three tasks for the cs1qa dataset. type classification task asks the model to predict the question type. code line selection task asks the model to select lines of code that are relevant to answering the given question. answer retrieval task finds a similar question already answered, and uses its answer as the answer to the given question. the outputs for these tasks can help the students debug their code and the tas spend less time and effort when answering the students' questions. finally, we implement and test baseline models, roberta @xcite , codebert @xcite and xlm-roberta @xcite , on the type classification and code line selection tasks. the finetuned models achieve accuracies up to 76.65% for the type classification task. the relatively low f1 scores of 57.57% for the line selection task suggest that the task is challenging for current language models. we use dpr @xcite to retrieve the most similar question and its answer. we compare the retrieved answer with the gold label answer, and achieve a bleu-1 score of 13.07, which shows incompetent performance of answer retrieval on cs1qa dataset. we show with a qualitative evaluation the model behavior with different inputs for the first two tasks. our contributions are as follows: ## related work code-based datasets recently, research dealing with large amounts of source code data has gained attention. often, the source code data is collected ad hoc for the purpose of the research @xcite @xcite . several datasets have been released to aid research in source code comprehension, and avoid repeated crawling and processing of source code data. these datasets serve as benchmarks for different tasks that test the ability to understand code. such datasets include: eth py150 corpus @xcite , codenn @xcite , codesearchnet @xcite and codeqa @xcite . we compare these datasets with cs1qa in in an educational setting, students' code presents different chracteristics from code in these datasets: 1) students' code is often incomplete, 2) there are many errors in the code, 3) students' code is generally longer than code used in existing datasets, and 4) questions and answers from students and tas provide important additional information. in cs1qa, we present a dataset more suited for the programming education context. ## task definition we design three tasks for the cs1qa dataset that identify important information that leads to the answer. the type classification task is to predict the question's type. we use nine types of questions that we categorized as the candidates for classification, each question belonging to a single type. we use the accuracy and macro f1 score as the measure of performance. the code line selection task is to select lines of code that give relevant information to answer the question. the code is a strong supporting context to answering the given question, and this task tests the model's ability to retrieve this critical information. for the code line selection task, we use the exact match (em) and line f1 score as the measure of performance, same as the metrics used for supporting fact selection task in hotpotqa @xcite . the em score measures the proportion of selections that exactly match the ground truth. the line f1 score measures the average overlap between the selected lines and the ground truth selections. the score is computed by treating the selections as bags of lines and calculating their f1 with the annotated lines. these two tasks take as inputs the lab and task numbers of the question, the questioner (student or ta), question and the code texts. the answer retrieval task retrieves a similar question given an unseen question, and uses the retrieved question's answer as the answer to the unseen question. bleu score is calculated between the retrieved answer and the gold label answer. answer generation task given the question and the code context is possible with cs1qa dataset. however, meaningfully generating the answer demands a model that understands long and erroneous code, and the natural language question. this poses a significant challenge, and we leave the generation task as future work. ## experimental setup we select three baseline models, codebert, roberta and xlm-roberta, and test their performance on the type classification and code line selection tasks. codebert model is selected to test the effectiveness of pretraining on nl-pl paired data. other models based on syntactic structures of code cannot take students' erroneous code as input. roberta and xlm-roberta models are selected to test the performance of nl-based models, for translated and untranslated data respectively. questions translated to english are provided to the two models pretrained in english, codebert and roberta. xlm-roberta model receives the untranslated questions as input to compare the performance when using the untranslated data. we used the default hyperparameters used in codebert for training. the tokenizers encode newline token to maintain the code's structure in the tokenized text. for the code line task, we also test the performance of the naive baseline, which selects the middle 60 lines of code, which showed the best performance among different numbers of lines, as the output. since the token lengths for code in cs1qa are greater than the limit of the transformer-based models, we preprocess the input to fit within the token length limit. we split the code into smaller segments so that the combined length of the split segment and the question is within the limit. for type classification, the type with the most number of votes is selected as the final selection. for code line selection, the model chooses a start and end token position from each segment. the lines between the start and end tokens are given as the output for the segment, and the union of segment outputs is given as the final selection for the question. n/a is given as the output when 1) the end position is before the start position, 2) either the start or the end position is 0 ([cls] token), or 3) either the start or the end position is out of range. for the answer retrieval task, we train the dpr by taking the questions as the passages. we use the question with the highest bm25 score in the corpus set as the gold label for the questions in the training set. for testing, the most similar question in the corpus is retrieved using the trained dpr with the new question as the query. the retrieved answer is used as the answer to the new question verbatim. ## results we report the mean score from three runs with different seeds for all experiments. the test score is reported on the best-performing epoch out of 10 on the development set. ## conclusion in this paper, we present cs1qa, a dataset for codebased question answering in introductory programming course. cs1qa's crowdsourced data from a programming course provide rich information that code understanding models need to consider to correctly answer the given questions. we introduce three tasks for cs1qa, whose output can help students debug and reduce workloads for the teaching staff. results from the baseline models indicate that tasks for cs1qa are challenging for current language understanding models. cs1qa promotes further research to better represent and understand source code for code-based question answering. as cs1qa data deliver the full context of the questions, the answer texts in cs1qa can be used as training and testing data for an answer generation task in the future. although the generation task is difficult and demands new code representation and processing methods, models that show good performance on it will allow a new level of automation in code-based qa. we hope that cs1qa will bring research interest in the domain of code understanding for question answering.