Pedagogical Alignment of Large Language Models (2024)

Abstract: Large Language Models (LLMs), when used in educational settings without pedagogical fine-tuning, often provide immediate answers rather than guiding students through the problem-solving process. This approach falls short of pedagogically best practices and limits their effectiveness as educational tools. We term the objective of training LLMs to emulate effective teaching strategies as ‘pedagogical alignment.’ In this paper, we investigate Learning from Human Preferences () algorithms to achieve this alignment objective. A key challenge in this process is the scarcity of high-quality preference datasets to guide the alignment. To address this, we propose a novel approach for constructing a large-scale dataset using synthetic data generation techniques, eliminating the need for time-consuming and costly manual annotation. Leveraging this dataset, our experiments with Llama and Mistral models demonstrate that LHP methods outperform standard supervised fine-tuning (SFT), improving pedagogical alignment accuracy by 13.1% and 8.7% respectively.Existing evaluation methods also lack quantitative metrics to adequately measure the pedagogical alignment of LLMs. To address this gap, we propose novel perplexity-based metrics that quantify LLMs’ tendency to provide scaffolded guidance versus direct answers, offering a robust measure of pedagogical alignment. Our analysis provides compelling evidence for the superiority of methods over SFT in optimizing LLMs’ behavior, underscoring the potential of methods in better aligning LLMs with educational objectives and fostering effective learning experiences. Code and models are available here.

## introduction in this paper, we propose a novel approach to achieve pedagogical alignment by modeling it as learning from human preferences (lhp). this method allows us to represent desired teaching behaviors as preferences, enabling more nuanced optimization than sft. however, a significant challenge in applying lhp is the scarcity of highquality, pedagogically-sound preference data. to address this, we introduce a key innovation: leveraging the structured output from the class framework @xcite to create meaningful preference pairs of pedagogically aligned and misaligned responses. this enables us to generate a large-scale, synthetic preference dataset that transforms abstract tutoring principles into concrete, comparative examples, allowing lhp algorithms to learn effective teaching strategies. next, using this preference dataset, we study the effectiveness of three lhp algorithms: direct preference optimization (dpo) @xcite , identity preference optimization (ipo) figure 1 : the image depicts the comparison between a traditional large language model (llm) interaction (left) and a pedagogically-aligned llm interaction (right). the traditional llm directly provides the user with the answer, while the pedagogically-aligned llm guides the student to the solution by presenting a series of subproblems. this elucidates the concept of pedagogical alignment, emphasizing the transformation from direct problem-solving to a guided, scaffolded learning experience. @xcite , and kahneman tversky optimization (kto) @xcite for pedagogical alignment of three open-source llms -llama @xcite , mistral @xcite and zephyr @xcite . we evaluate the llms using pedagogical alignment accuracy and f1 score which measure if the models produce the desired pedagogical responses. our results show that dpo and kto significantly improve llama, mistral and zephyr's performance, improving pedagogical alignment accuracy by 13.1%, 8.7% and 50.0%, respectively, compared to sft. existing evaluation methods for llms in education also fail to adequately quantify pedagogical alignment. to address this gap and compare lhp with sft, we introduce a second preference dataset generation technique and a novel perplexity-based evaluation approach. this allows us to quantitatively measure an llm's tendency to provide stepby-step guidance versus direct answers, offering a robust metric for pedagogical alignment. using our technique we generate pedagogically misaligned actions, which we then compared to the aligned actions in terms of perplexity. our analysis revealed that base models are more inclined to provide direct solutions than offer hints and guidance. while sft partially corrects this behavior, it doesn't fully optimize it. however, lhp dramatically optimizes this behavior, promoting guidance over direct solutions. this shift in behavior underscores the effectiveness of lhp in aligning llms with educational objectives, providing compelling evidence for its superiority over sft in fostering effective learning experiences. ## preference data generation for pedagogical alignment in this section, we describe our dpo, ipo, and kto compatible data generation pipeline which we also outline in figure 2 . the objective is to generate a preference dataset d p which necessitates distinguishing between pedagogically chosen and rejected tutor responses based on their alignment with desired scaffolding strategies. d p will mimic a dataset structure that is compatible with dpo, ipo, and kto to optimally represent these preferences. this dataset has three fundamental components: context, chosen, and rejected. in our case, the context represents the accumulated conversational history between the student and the tutor leading up to a particular interaction point, providing the necessary backdrop for the tutor's next dialogue turn. the chosen contains the pedagogically preferred or optimal tutor responses to the context, while rejected includes those tutor responses considered less effective or misaligned with the scaffolding approach. conversational dataset d c and agents: we use the conversational prompt listed in to generate the preference dataset d p from d c , we employ three autonomous agents: gpt-student (a g ), gpt-tutor (a t ), and sft-tutor (a s ). the conversation starts with gpt-student a g posing a question (q) sourced from a question bank to which both gpt-tutor a t and sft-tutor a s provide responses in the following json structure: defining context: we define the context c t for any given response at time t as all preceding conversational turns up to that point: preference dataset d p construction: given this understanding of context, d p is structured to encapsulate interactions where the pedagogically preferred (chosen) response diverges from the alternative (rejected) based on specific pedagogical criteria. this divergence is identified by a function f that assesses the pivotal attributes of each response. hence, d p is defined as: the evaluation of these attributes by f facilitates an objective basis to deem one response as chosen (preferable) and another as rejected (less aligned with desired pedagogical outcomes) through a systematic assessment of their pedagogical value and alignment. ## experiments in this section, we provide a comprehensive overview of our dataset construction, experimental design, and the subsequent findings derived from the application of state-of-the-art lhp algorithms to train pedagogically-aligned llms. ## conclusion in this paper, we have investigated the application of learning from human preferences (lhp) methods to align llms with educational goals, aiming to foster optimal student learning outcomes. by constructing a preference dataset specifically designed for pedagogical alignment, we have laid the foundation for enhancing the effectiveness of llms in educational contexts. our experimental results, derived from applying lhp alignment algorithms on state-of-the-art open-source llms, demonstrate the superiority of lhp methods over standard supervised fine-tuning (sft), significantly improving the alignment of llms with pedagogical objectives. our another key contribution is the development of a novel approach for evaluating the pedagogical effectiveness of llms. by curating a preference dataset that compares the perplexity of responses offering scaffolding versus those providing direct solutions, we have introduced a new methodology that quantifies the extent to which llms prioritize pedagogically effective actions. this approach opens up new possibilities for assessing and optimizing the performance of llms in educational settings, providing a valuable tool for researchers and practitioners alike.