CLASS : A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles (2023)

Abstract: We present a design framework called Conversational Learning with Analytical Step-by-Step Strategies (CLASS) for building advanced Intelligent Tutoring Systems (ITS) powered by high-performance Large Language Models (LLMs). The CLASS framework empowers ITS with two key capabilities. First, through a carefully curated scaffolding dataset, CLASS equips ITS with essential problem-solving strategies, enabling it to provide tutor-like, step-by-step guidance to students. Second, by using a dynamic conversational dataset, CLASS assists ITS in facilitating natural language interactions, fostering engaging student-tutor conversations. The CLASS framework also provides valuable insights into ITS’s internal decision-making process which allows seamless integration of user feedback, thus enabling continuous refinement and improvement. We also present a proof-of-concept ITS, referred to as SPOCK, which is trained using the CLASS framework with a focus on introductory college level biology content. A carefully constructed protocol was developed for SPOCK’s preliminary evaluation, examining aspects such as the factual accuracy and relevance of its responses. Experts in the field of biology offered favorable remarks, particularly highlighting SPOCK’s capability to break down questions into manageable subproblems and provide encouraging responses to students.

## introduction intelligent tutoring systems (its) have a rich history of offering valuable support to students and educators, with successful implementations such as cognitive tutor in mathematics @xcite and autotutor for computer literacy @xcite . however, the development of effective its remains a challenge, particularly in addressing the diverse learning needs of students and promoting a deeper understanding of complex concepts. drawing on the potential of recent advancements in natural language processing, chat-based large language models (llms) such as chatgpt @xcite present an opportunity to build upon the existing its and further improve its by integrating llms with learning science principles @xcite . the application of learning science principles is crucial for developing its that effectively supports learners in their cognitive processes, provides personalized assistance, and fosters engaging learning experience @xcite . in this study, we present a novel design framework called conversational learning with @xcite that integrates these principles to create an effective language model-based its for biology, referred to as spock 1 . the core objective of the class framework is to equip its with two important capabilities: 1) providing tutor-like step-by-step guidance that fosters learners' deeper understanding 2) engaging learners in tutor-like conversations using natural language to ensure conversational adaptability. class utilizes two specifically curated training datasets to instill the desired capabilities in spock while aligning with learning science principles. the first dataset, "scaffolding dataset", is grounded in problem decomposition and scaffolding learning principles @xcite . this dataset covers essential components such as problems, related subproblems, hints, incorrect solutions, and customized feedback. the second "conversational dataset" builds on the foundation established by the scaffolding dataset and focuses on simulated conversational student-tutor interactions inspired by the socioconstructivist model of learning @xcite . the conversations, generated by gpt-4, incorporates figure 1 : a demonstration of class framework and spock's training process. the framework utilizes two synthetic datasets with distinct objectives to create its. the first scaffolding dataset aims to equip spock with step-by-step problem-solving skills. this dataset consists of problems, corresponding subproblems, hints, incorrect student responses and corresponding feedback. the second conversational dataset has an objective of helping spock apply these skills effectively in real-time conversations with students. this dataset contains simulated mock interactions between students and an ai tutorbot. both datasets are created using gpt-4 and a brief description of the specifically designed prompt instructions and outputs are displayed in the figure. class framework also uses an indexed search over related educational contents to reduce hallucination and maintain factual consistency during conversations. in the top part, we also present an example of interaction between students and spock. within the conversations contained in the second dataset, a pre-defined response template is employed to ensure consistency and coherence in spock's responses across various situations. this structured approach facilitates seamless user feedback incorporation and system enhancement by offering insights into spock's internal decisionmaking mechanisms for continuous improvement and refinement. our contributions can be summarized as follows: ## background in this section, we first provide an overview of its, then emphasize the influence of llms in the its designing. additionally, we highlight the fundamental principles of learning science that have motivated our design framework. ## training spock in this section, we provide the implementation details of spock using proposed class framework as a proof-of-concept. spock is built upon a powerful 13 billion parameter vicuna model @xcite . vicuna-13b is an open-source language model trained by fine-tuning the llama model @xcite on 70k user-shared conversations collected from the sharegpt website. we chose vicuna-13b because of its ability to generate detailed and well-structured answers compared to other open-source language models, such as alpaca @xcite . additionally, vicuna-13b has an elo-rating of 1061 which is highest among the 13 billion open-source llms on llm-as-a-judge chatbot arena @xcite . to provide spock with domain-specific knowledge, we further fine-tuned the vicuna-13b model on 60 libretexts biology textbooks @xcite using the causal language model (clm) loss with the help of the huggingface transformers library @xcite . this fine-tuning step aims to enhance spock 's understanding of biology concepts, as the vicuna-13b model attains a relatively low score on the mmlu benchmark @xcite when responding to questions in the stem and social sciences domains. following the clm fine-tuning, we created the two datasets that form the backbone of the class next, we created the conversational dataset by prompting gpt-4 to generate mock conversations between a student and an ai-tutorbot by using the problems from the scaffolding dataset. this dataset contains 648 conversations summing up to a total of 20k student-tutorbot interactions. average length of conversations is around 400 words, only including the student and tutorbot fields in the conversation template. once the two datasets were generated, we further trained the vicuna-13b model on both datasets with the help of the deep-speed @xcite and fastchat @xcite libraries. the cost of training spock can be broken down into two primary components. first, the creation of both datasets involves prompting gpt-4, which costs approximately $50 each. second, we train the model using the clm loss on 60 biology textbooks and then fine-tune it on both scaffolding and conversational datasets for 10 epochs each. this process is executed on 8 nvidia rtx 48-gb a6000 gpus and runs for three days. in summary, the implementation of spock involves model selection, domain-specific fine-tuning, class datasets generation, and further model fine-tuning. ## evaluation in this section, we begin with a human evaluation to assess the quality of our synthetic scaffolding datasets. we engaged four subject matter experts (smes) who possess graduate-level knowledge in biology. subsequently, we propose an evaluation protocol for its based on class framework and proceed to conduct a preliminary evaluation of spock. for this evaluation, we collaborate with an educator at an anonymized college along with three senior graduate-level biology students.