B ook2 D ial: Generating Teacher Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots (2024)

Abstract: Educational chatbots are a promising tool for assisting student learning. However, the development of effective chatbots in education has been challenging, as high-quality data is seldom available in this domain. In this paper, we propose a framework for generating synthetic teacher-student interactions grounded in a set of textbooks. Our approaches capture a key aspect of learning interactions where curious students with partial knowledge interactively ask teachers questions about the material in the textbook. We highlight various quality criteria that such dialogues must fulfill and compare several approaches relying on either prompting or finetuning large language models according to these criteria. We use the synthetic dialogues to train educational chatbots and show the benefits of further fine-tuning in educational domains. However, careful human evaluation shows that our best data synthesis method still suffers from hallucinations and tends to reiterate information from previous conversations. Our findings offer insights for future efforts in synthesizing conversational data that strikes a balance between size and quality. We will open-source our data and code.

## introduction educational chatbots offer a scalable way to improve learning outcomes among students @xcite . however, building educational chatbots has been challenging as high-quality data involving teachers and students is difficult to obtain due to various practical reasons such as privacy concerns @xcite . in response to this, we study the task of generating synthetic teacherstudent interactions from textbooks. we create a novel dataset of textbooks drawn from an open publisher of student textbooks and present a framework ( book2dial) to generate synthetic teacherstudent interactions from these textbooks. our teacher-student interactions take the form of conversational question-answering (qa) interactions @xcite where curious students ask teachers questions about the textbook and teachers answer these questions based on the textbook. our approach primarily extends the direct instruction phase of teaching factual knowledge by facilitating more interactive information exchanges with student @xcite . this differs from techniques focusing on teaching using scaffolding @xcite or socratic questioning @xcite which leads to deeper and wider human learning. however, the task of generating high-quality synthetic dialogue data is difficult @xcite . this is amplified in education, where the generated interactions should cover the teaching material in an informative and coherent way @xcite . therefore, it is important to have quality controls on such data, because students might otherwise receive wrong feedback, which could be detrimental to learning. thus, in this work, we also formulate some criteria that measure the quality of educational dialogues. for example, it is crucial that the chatbot does not provide students with incorrect information and stays grounded in the textbook, ensuring factual consistency with the knowledge taught. this is particularly important given that large language models (llms) are prone to 'hallucinations' or generating plausible but incorrect or unverified information @xcite . while a simple teacher strategy would be to answer with extracted passages from the textbook, this might hurt the coherence of the dialogue which is present in interactive educational situations @xcite . the teacher's response should both be relevant to the student's question @xcite , as well as, informative as this ensures that key information from the textbook is covered in the dialogue @xcite . we formalize these requirements into figure 1 : example of a synthetic teacher-student interaction based on a textbook, along with various criteria for evaluating the quality of the interaction. the criteria include answer relevance of the answer to the question, coherence of the question-answer interaction with respect to the dialogue history, informativeness of the overall interaction, groundedness to the textbook, answerability of the question from the textbook, factual consistency of the answer with respect to the question, and specificity of the question. more details in section 3.2. criteria shown in figure 1 . book2dial comprises of three approaches taken from prior work: multi-turn qg-qa @xcite , dialogue inpainting @xcite and prompting llms to role-play teacher and student. we use the formatting information in the textbook, such as titles or key concepts, to initialize student models with imperfect information. in contrast, the teacher models have perfect information and are expected to generate grounded responses based on the textbook. we fine-tune and prompt various open-source language models to generate teacher-student interactions. we evaluate book2dial on the proposed quality criteria and use human evaluations to support our findings. results reveal that data generated by role-playing llms scores highest in most criteria, as shown in section 5.1.1 and 5.1.2, demonstrating reasonable efficacy in creating educational dialogues. the generated dialogues effectively incorporate textbook content but fall short in mimicking the natural scaffolding of educational conversations and exhibit issues like hallucinations and repetition, as discussed in section 5.3. despite these limitations, we show that the generated synthetic data may be used to fine-tune educational chatbots with benefits in some educational domains (section 5.4), especially when the domain of the evaluation dataset matches with the fine-tuning data. datasets like quac @xcite , coqa @xcite and ultrachat @xcite focus on conversational question answering in non-educational settings. previous work has also explored strategies for creating such data with humans or automatically by using models. for example, @xcite withholds the context required for answers from the questioner, leading to information-seeking questions. simseek @xcite synthesizes datasets for conversational question answering from unlabeled documents but does not demonstrate significantly improved performance in downstream tasks. a recent work, dialogizer @xcite , proposes a framework for generating context-aware conversational qa dialogues. however, these methods do not take into account the specifics of the educational domain. other works outside of conversational question answering also explore data synthesis. book2dial framework for generating dialogues from textbooks: our approach uses two models -a student model and a teacher model. the student model plays the role of a student, formulating questions from a limited context (document formatting). in contrast, the teacher model assumes the role of a teacher, providing answers and guidance by referencing the (sub-)section in the textbook. this framework can be adapted to various instantiations of the two roles with varying formatting information, such as multi-turn qa-qg models @xcite , dialogue inpainting @xcite , and a prompting approach utilizing role-playing llms. grounded dialogue datasets. ## educational conversation generation we first introduce a framework for dialogue synthesis from textbooks in section 3.1, and then discuss the quality criteria that the generated dialogues should fulfill in section 3.2. ## from textbooks to dialogues in this section, we describe different methods used for generating dialogues from educational textbooks in book2dial, namely: 1. multi-turn qg-qa models. in this setting, we use fine-tuned qg and qa models interacting with each other. we describe these methods next. more implementation details are in appendix c. ## results and analyses in this section, we aim to address the following research questions: to address these questions, we generate dialogues from textbooks across various domains and analyze the generated dataset. textbook data: we collect 35 textbooks available on openstax foot_2 , spanning domains of math, business, science, and social science. from these, we select four textbooks to create our dialogue datasets. ## conclusion this work introduces a new task of generating conversational question-answering dialogues from textbooks to help fine-tune educational chatbots in various domains where high-quality dialogue data are scarce. we detail and compare various approaches and settings to simulate student-teacher interactions and create such data. we evaluate the generated dialogues, focusing on some measures of their quality, such as answer relevance, informativeness, coherence, and factual consistency. our results indicate that the approach with llms role- book2dial can be used to fine-tune educational chatbots and achieve performance improvements in various educational settings.