Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations (2023)

Abstract: Socratic questioning is a teaching strategy where the student is guided towards solving a problem on their own, instead of being given the solution directly. In this paper, we introduce a dataset of Socratic conversations where an instructor helps a novice programmer fix buggy solutions to simple computational problems. The dataset is then used for benchmarking the Socratic debugging abilities of GPT-based language models. While GPT-4 is observed to perform much better than GPT-3.5, its precision, and recall still fall short of human expert abilities, motivating further work in this area.

## introduction and motivation educational needs for computer science (cs) are on the rise, due to increased enrollments in cs programs @xcite . higher education institutions in particular are affected by the lack of sufficient instructional staff, often resorting to hiring undergraduate teaching assistants (tas) in their computer science courses. an effective ta benefits students by providing timely feedback and assistance that is tailored to each student's level of proficiency, with measurable and significant impact on student retention rates @xcite . in practice, however, not all educational institutions benefit uniformly from their tas. depending on class sizes and ta allocations, it is often the case that a teaching assistant cannot spend their time equally with all students who need help, especially when nearing office hours or an assignment deadline. moreover, students who lack fundamental knowledge from prerequisite courses consume significant ta time throughout the course. this comes at a time when there is also a shortage of k-12 computer science teachers, a lack of appropriate training for k-12 educators interested in teaching cs effectively @xcite , and rising ta and peer instruction demand in flipped computer science classrooms @xcite . overall, the lack of instructional staff, ranging from tas to k-12 teachers and college educators, motivates the automation of various types of teaching tasks by leveraging the increasing capabilities of ai models, especially in terms of understanding and generating language and code. prior work in ai for programming education is primarily composed of intelligent tutoring systems (its) and learning support systems for programming courses. while some its systems allow interactions with a learner through a chat interface @xcite , the range of interactions is often limited, as tutoring systems typically focus on giving hints constructed for predefined solutions or predefined socratic utterances that are specific to a known set of programming exercises @xcite @xcite . consequently, traditional its systems in the programming domain do not generalize to new courses or new coding assignments without human intervention. this situation is however rapidly changing, due to the substantial leaps in performance exhibited by large language models recently, on a wide array of problems. language models are now capable of solving introductory programming exercises @xcite including custom problems created by instructors @xcite . furthermore, solutions generated by these models are unique and can fool plagiarism software such as moss @xcite , presenting educators with further challenges in maintaining academic integrity. in socratic questioning, a teacher assists a learner trying to solve a problem beyond their zone of proximal development @xcite . language models (lms) have been used effectively for generating a particular type of socratic questions for solving word math problems, wherein they leverage the sequential structure of steps that compose the solution @xcite . other applications of lms include automated feedback on student code submissions @xcite , as well as generating programming exercises, unit tests, and code explanations @xcite . however, there still remains a substantial gap in leveraging lms effectively for guiding novice programmers through a coding exercise in a way that maximizes their learning outcomes, similar to how an effective, experienced ta would guide a beginner programmer. for socratic questioning, in particular, the difficulty of building an effective system is compounded by the scarcity of examples, whereas the limited data that can be found @xcite does not have sufficient structure to enable the automatic evaluation of socratic questioning systems. in this paper, we focus on the task of socratic questioning for debugging @xcite , or socratic debugging, defined as a conversation between a knowledgeable programmer and a beginner student who comes for help fixing a buggy solution for a simple computational problem (section 2). to enable the development and evaluation of lmbased instructional agents, we introduce a manually created dataset of dialogues where the main objective is for the student to repair their buggy code themselves by leveraging guidance received from the instructor at every turn (section 3). however, as originally observed by @xcite , "no precise formula, or line of questioning" is needed to achieve the goals of socratic questioning. furthermore, depending also on their expectations with respect to the student's abilities, an instructor can often think of multiple ways of guiding the student at any particular turn in the conversation, leading to a very large space of possible dialogues. socratic questions lie in a continuum ranging from providing direct hints that give out the answer to offering minimal guidance, enabling instructors to pose queries at an appropriate level that challenges the student while remaining within each student's ability to answer. to facilitate the automatic evaluation and benchmarking of future socratic questioning systems in terms of their precision and recall, the dataset contributors are asked to provide all alternative utterances that they think could help the student, at every turn in the conversation. this is a currently ongoing, cognitively demanding data generation effort, requiring contributors with sub-stantial experience in tutoring beginner programmers. we use the current version of the dataset, containing 86 main conversations, to benchmark the socratic debugging abilities of two large language models in the gpt family, namely gpt-3.5 and gpt-4 (section 4), noticing a large discrepancy in performance in favor of the more recent gpt-4. we conclude the paper with related work and limitations. ## task definition we formulate the socratic debugging task as a dyadic conversation between a student and an instructor. in this scenario, the student is assumed to be a beginner programmer who has recently started learning how to code in python. as part of his foot_0 learning to code curriculum, the student is given a coding problem for which he needs to write a function implementing the specified input-to-output relationship. the student writes the code for the function, however, the code is buggy and he cannot make progress on his own without help, therefore he seeks help from the instructor. the instructor is assumed to be a proficient programmer in python with experience in teaching novice programmers how to code. when contacted by a student for help, her main aim is to maximize the learning outcomes by following a socratic guidance approach through which, over one or more dialogue turns, she helps the students figure out where the bug is and how to fix it on their own. ## benchmark dataset to facilitate the development of conversational agents that act under the task definition above, we manually created a dataset of dialogues where a student fixes buggy code on his own by leveraging the socratic guidance received from an instructor. the dataset is created by sequentially specifying the coding @xmath0. first, a coding problem is selected, normally a simple coding exercise situated at a novice level of coding proficiency, such as factorial or fibonacci. the coding problem is specified through the problem description and the associated test cases. next, one or more buggy implementations are created, with the constraint that each implementation contains exactly one bug. the bugs were selected to reflect common types of mistakes that beginner programmers make, such as forgetting that indexing of sequences starts at 0, boundary bugs, operator misuse, or misunderstanding of basic programming constructs. for each buggy implementation, a main conversation is created, where a fictional student, the author of the buggy code, interacts with a fictional instructor. the aim of the instructor is to guide the student to discover the cause of the bug and fix it on his own through socratic dialogue. the dialogue always starts with a student utterance. the instructor and the student then take turns in a dialogue, until the bug is successfully fixed. at each turn, the student may also provide a block of code if he made edits to the code at that turn. following research in dialogue systems @xcite , we create multiple reference instructor utterances at each turn. the main utterance may be optionally followed by one or more alternative utterances. given that the aim of this dataset is to benchmark the ability of an artificial instructor agent to generate socratic guidance, it is especially important that the contributed main and alternative utterances for the instructor comprehensively explore the entire range of socratic advice at that point in the conversation. these alternative utterances should be semantically distinct in a non-trivial manner; in particular, they should not be mere paraphrases of the main utterance or of each other. upon inspection of the conversations created manually, we discovered that one contributor used a vending machine as an analogy to guide the user to conclude that print is not the same as return. while using analogies can substantially enhance the impact of socratic questioning, it can lead to an open-ended range of alternatives, as the number of possible analogies is virtually infinite. since our aim is to create a dataset that can be used to estimate both the recall and precision of a socratic guidance generator, at this stage we decided to require that socratic utterances be literal, leaving the generation of figurative utterances as a direction for future work. for the student, alternative utterances may give different or conflicting answers to an instructor question, reflecting different levels of understanding. students may give correct or incorrect answers; they may also introduce new bugs when fixing the original bug. once the main conversation ends with the student successfully correcting their code and passing all test cases, the contributors are instructed to create up to three conversational threads. the dialogues in the dataset were created by 10 contributors with extensive experience in cs education as instructors, teaching assistants, or tutors. the starting problems and buggy implementations were selected to contain a variety of syntactic to streamline and standardize the collection of socratic dialogues and code edits for each input problem description and buggy implementation, we developed a 7-page web application using the streamlit foot_1 and gsheetsdb foot_2 libraries. the application guides contributors through selecting a bug, creating initial and conversational threads, and reviewing and submitting their work. during the process, contributors can add main and alternative utterances, undo actions, and edit the chat history. the application also allows importing and exporting dialogues in a standardized form for review. for more details and images of the application, the reader is referred to appendix a. ## experimental evaluations we evaluate the gpt-3.5 (openai, 2022) and gpt-4 (openai, 2023) language models in terms of their capacity to generate, at each instructor turn, socratic utterances that match those contributed in the benchmark dataset. each test example is composed of an input prompt to the language model containing: a steering prompt for socratic questioning adapted from the gpt-4 blog post foot_3 , the problem description, the buggy code, the bug description, the bug fixes, the unit tests, the dialogue history so far, and an instruction to the language model to generate all possible semantically distinct socratic utterances, as shown below. respond to the user with all possible distinct socratic utterances that guide the user to discover and fix the bug described between '<@xmath1. student code is written between '<@xmath2. utterances that have the same meaning but different words are considered duplicates. assume that the student has run the test cases. the list of utterances generated by the lm is then used to estimate precision and recall. after conducting a preliminary, qualitative evaluation of various prompts and instructions we select the prompt and instruction used in this paper. for more details about prompting, the reader is referred to appendix b. in all experiments, lm outputs are generated using a greedy decoding setting (i.e. @xmath3. we set a maximum generated token threshold of 1,024 and do not apply any frequency or presence penalties. we perform manual evaluation of the lm generations for a subset of problems, and automatic evaluations for all problems in the benchmark dataset. ## conclusion & limitations this paper presents a dataset of expert-curated socratic conversations where instructors assist novice programmers in fixing buggy solutions to simple computational problems. the dataset serves as a benchmark for evaluating the socratic debugging capabilities of lms. while gpt-4 outperforms gpt-3.5, its precision, and recall remain below human expert levels (70.0), highlighting the need for further research. we find that gpt-family language models may generate repetitive and irrelevant socratic utterances that could mislead learners. the utterances may also appear too early in the conversation, causing confusion, and can be overly direct, potentially diminishing learning outcomes. study limitations include: the automatic metrics are limited in capturing the correctness, helpfulness, and relevance of a socratic utterance, and the benchmark dataset may not represent all common novice misconceptions. moreover, the manual evaluation is limited to 5 dialogues and could be expanded, but this process is highly time-consuming.