Improving Socratic Question Generation using Data Augmentation and Preference Optimization (2024)

Abstract: The Socratic method is a way of guiding students toward solving a problem independently without directly revealing the solution to the problem by asking incremental questions. Although this method has been shown to significantly improve student learning outcomes, it remains a complex labor-intensive task for instructors. Large language models (LLMs) can be used to augment human effort by automatically generating Socratic questions for students. However, existing methods that involve prompting these LLMs sometimes produce invalid outputs, e.g., those that directly reveal the solution to the problem or provide irrelevant or premature questions. To alleviate this problem, inspired by reinforcement learning with AI feedback (RLAIF), we first propose a data augmentation method to enrich existing Socratic questioning datasets with questions that are invalid in specific ways. Also, we propose a method to optimize open-source LLMs such as LLama 2 to prefer ground-truth questions over generated invalid ones, using direct preference optimization (DPO). Our experiments on a Socratic questions dataset for student code debugging show that a DPO-optimized LLama 2-7B model can effectively avoid generating invalid questions, and as a result, outperforms existing state-of-the-art prompting methods.

## introduction learning based on a conversation that consists of questions and answers, where the student responds to questions posed by a more knowledgeable instructor, has been proven to be effective in teaching students about a particular concept @xcite . in particular, socratic questioning, which refers to a way for the instructor to guide a student to solve a problem (within their zone of proximal development) by asking them questions that pro-mote thinking while not directly revealing the solution @xcite , is a very effective pedagogical method in conversation-based learning and tutoring. recent advances in large language models (llms) @xcite have led to the rapid development of chatbots that promote student learning by automatically generating the instructor's utterances @xcite @xcite . one key area of interest in the development of such chatbots is question generation, which can help students solve logical problems in the mathematics and programming domains @xcite . typically, question generation in educational applications has focused on generating practice or assessment questions, in biology exams @xcite , reading comprehension @xcite , math practice @xcite , and programming exercises @xcite . as a specific form of question generation, socratic question generation has gained attention, owing to its effectiveness in improving student learning outcomes by eliciting critical thinking and self-discovery during problem-solving @xcite . socratic questions generation is a complex task because it involves mapping out the step-bystep thought process of students during problemsolving, locating the cause of their error, and providing effective questions without revealing the solution. manually generating socratic questions can be a cognitively demanding and time-consuming task for instructors. several recent works proposed to automatically generate socratic questions using llms: in math education, @xcite shows that generating a sequence of socratic subquestions and prompting students to answer helps them solve math word problems more successfully. in computer science education, @xcite releases a dataset on socratic questions for student code debugging and provides baselines based on llm prompting and finetuning. in particular, the authors prompt gpt-3.5-turbo and gpt-4 @xcite in a chain-of-thought manner @xcite to generate socratic questions. a human study by the authors shows that the generated questions can sometimes be invalid in several different ways, including being irrelevant to the problem, repetitive of earlier dialogue turns, or too direct and revealing the solution prematurely, which may hamper students' learning processes. since gpt models are proprietary and expensive, the authors also attempt to fine-tune the open-source flan-t5 model @xcite ; however, doing so proves to be ineffective due to its insufficient scale and the pretraining procedure used. in this paper, we propose a method to improve the validity of automatically generated socratic questions using open-source llms. our method is inspired by recent developments in reinforcement learning with ai feedback (rlaif) @xcite ; our method consists of two phases, data augmentation and preference optimization. specifically, our contributions are as follows: ## problem definition and dataset we study the problem of socratic question generation in conversations between a student and an instructor, where the instructor's goal is to guide the student through the process of solving a problem. concretely, our goal is to generate socratic questions at a particular dialogue turn for the instructor during the conversation, given the dialogue history and contextual information about the problem the student is trying to solve and their solution. in this work, we use the dataset for code debugging introduced in @xcite . the dataset is based on didactic conversations between a student and an instructor, where the student is a novice programmer tasked with writing a program for a given problem. the dataset consists of the student's buggy code submissions along with a dialogue between the instructor and the student, where the instructor asks socratic questions in the form of a conversation to help the student debug their code. the conversation consists of dialogue turns with each instructor utterance being a collection of several possible "ground-truth" socratic questions at that dialogue turn. the dataset also contains metadata including the problem statement, the test cases, the bug description, and code fixes to resolve the bug. in total, there are 38 problems with more than 50 different bugs in student solutions, and conversations centered around these buggy codes containing more than 1900 dialogue turns. the dataset is split into two subsets, a train set and a test set which contain 135 and 16 dialogues, respectively, spread across different problems. ## proposed method in this section, we describe our method for the task of socratic question generation. our method involves two phases: first, data augmentation, and second, preference optimization, as shown in figure 1. ## results and discussions in the consistency checking step of the data augmentation phase, we see that 72% of the generated questions are considered for the preference dataset creation as 27% of the generated questions are classified as "good" and 1% as "incorrect". this result shows that gpt-4 is more prone to generate "good" questions for particular dialogue turns than incorrect questions that do not relate to the problem and the dialogue. for the task of socratic question generation, for the baseline methods that use zero-shot llama prompting, we observe that llama (cot) is the best, which shows that chain-of-thought prompting to first generate the students' current misconceptions followed by the socratic questions is effective. among the preference optimization experiments, we see that dpo consistently outperforms sft. we also observe that the llama (cot) performs as well as dpo greedy in terms of bertscore f1 as llama (cot) generates a higher number of socratic questions whereas the dpo greedy method just generates one. hence, the recall of the dpo greedy method is lower than that of llama (cot). among decoding variants, we see that the sample-5 method is better than the greedy method highlighting the importance of sampling multiple possible socratic questions instead of just one. overall, we see that our preference-optimized models with dpo give the best rouge-l scores for all precision, recall, and f1 scores with dpo greedy having the highest precision and dpo sample-5 having the highest recall and f1 score among all the methods. dpo greedy has the highest bertscore precision, whereas dpo sample-5 has a recall comparable to the best gpt method, gpt-4 (cot). these results suggest that the dpooptimized llama 2-7b model is better than (or as effective as) much larger models like gpt-4 (25 times larger) for socratic question generation. ## case study we now use a case study to illustrate why our method leads to better socratic question generation. first, we show an example of invalid socratic questions generated by our data augmentation phase. second, we compare different methods for socratic question generation. the second question, which asks the student to examine the value of fact is valid since it does not directly reveal the bug. in contrast, most of the dpo sample-5 questions are valid, since they urge the student to examine the value of i and fact on lines 5 and 6 with specific values of n, without directly revealing the bug that i is always 0. through these comparisons, we see that dpo improves socratic question generation compared to sft and that dpo sample-5 is highly capable of generating valid yet diverse questions. ## conclusions and future work in this work, we propose a method for socratic question generation in programming problem feedback scenarios. our method consists of a data augmentation phase to create a preference dataset by synthetically generating invalid questions according to four possible categories. we then use this preference dataset to fine-tune an opensource llm, llama 2-7b, using direct preference optimization (dpo). our results show that the preference-optimized llama 2-7b model often outperforms existing state-of-the-art prompting methods (on common text similarity metrics) that rely on much larger gpt models (25 times larger), by avoiding invalid questions after training on the augmented dataset. our method paves the way toward an open-source, accessible, cheaper, privacypreserving, yet effective alternative to generating socratic questions which can improve students' learning outcomes without having to rely on proprietary rate-based api-accessed models like gpt-4. there are several avenues for future work. first, we can develop a technique to differentiate types of invalid socratic questions and not treat them equally while performing preference optimization. this technique would require us to modify the inherent objective function of dpo to incorporate more than one unpreferred question for a single preferred question, which may give us fine-grained control over the llm generations. second, we can experiment with open-source llms that are larger than 7b to see whether dpo provides more significant gains over sft on larger models on the socratic question generation task. third, we can perform a systematic human evaluation to compare the performance of our proposed method with other baselines. also, we can focus on designing an automatic metric (based on llm prompting (liu et al., 2023)) other than rouge and bertscore which captures the helpfulness of the socratic questions without heavily relying on assigning higher scores only to questions that have high lexical overlap with the ground-truth questions. fourth, we can experiment with alternative preference optimization methods, such as kto (ethayarajh et al., 2024) which do not need explicit preference data in the form of pairs of valid and invalid questions. fifth, we can also explore if socratic question generation helps in improving other tasks in computer science education like test case generation (kumar and lan, 2024) by posing the problem as answering several socratic sub-questions (shridhar et al., @xcite . finally, we can also explore how to make socratic question generation knowledge-aware, i.e., generating different questions for students with different knowledge states, which can be estimated using the open-ended knowledge tracing method for computer science education @xcite .