Curriculum-Driven Edubot: A Framework for Developing Language Learning Chatbots through Synthesizing Conversational Data (2024)

Abstract: Chatbots have become popular in educational settings, revolutionizing how students interact with material and how teachers teach. We present Curriculum-Driven EduBot, a framework for developing a chatbot that combines the interactive features of chatbots with the systematic material of English textbooks to assist students in enhancing their conversational skills. We begin by extracting pertinent topics from textbooks and using large language models to generate dialogues related to these topics. We then fine-tune an open-source LLM using our generated conversational data to create our curriculum-driven chatbot. User studies demonstrate that EduBot outperforms ChatGPT in leading curriculum-based dialogues and adapting its dialogue to match the user’s English proficiency level. By combining traditional textbook methodologies with conversational AI, our approach offers learners an interactive tool that aligns with their curriculum and provides user-tailored conversation practice. This facilitates meaningful student-bot dialogues and enriches the overall learning experience within the curriculum’s pedagogical framework.

## introduction the emergence of conversational agents has significantly impacted educational technology, changing how students interact with material and how teachers impart knowledge @xcite @xcite . these agents, more commonly known as "chatbots," have shown usefulness in various educational settings, from teaching computer programming @xcite to strengthening conversational skills @xcite . however, its application comes with inherent challenges, especially in conversational skill development. most chatbots primarily respond to user queries and follow the instruc-* both authors contributed equally to the work. to address these challenges, we propose a framework called curriculum-driven edubot for developing a chatbot based on a specific curriculum. our chatbot will focus on predetermined topics and use vocabulary from the curriculum to better align with the english proficiency of the users. it will act as a conversational practice partner, combining the interactive features of chatbots with the organized content of english textbooks. first, we extract relevant topics from textbooks and use large language models (llms) to synthesize fixed-format personas for both participants in the dialogue. then, we use llms to synthesize dialogues based on these topics and personas, incorporating the vocabulary provided in the textbook. subsequently, we fine-tune an open-source model with our generated conversational data to construct our chatbot. our chatbot is more than just a responsive tool, it is an academic companion that guides students through coherent and friendly dialogues tailored to their english proficiency level. as illustrated in figure 1 , existing chatbots, such as chatgpt, are not based on a curriculum. instead of being conversational learning partners, they mainly act as ai-driven q&a systems, and their content may not align with the student's educational objectives. in contrast, our chatbot is constructed from synthesized dialogues that include clearly defined characters, curriculum-appropriate topics, and textbookbased vocabularies, thus providing an interactive and user-tailored conversational experience. we conducted a thorough user study to evaluate curriculum-driven edubot, using a high-quality college english textbook intended for english learners as a benchmark. our findings indicate that our chatbot outperforms chatgpt in various metrics. specifically, 75% of students found edubot to be particularly effective in facilitating interactive conversations, and they believed it was better suited to their english proficiency. the results and conversation examples from the user study clearly demonstrate that our chatbot is more closely aligned with the role of a language-learning companion. furthermore, 83.3% of students expressed willingness to recommend edubot to others, and 87.5% of students believe that interactions with edubot can help students improve their conversational skills. in summary, our main contributions are as follows: ## related work many studies have shown that artificial intelligence (ai) can be utilized in educational settings @xcite @xcite . for example, rodrigues language learning, traditionally dependent on static resources such as textbooks and structured courses, has benefited greatly from curriculumaligned approaches that combine consistency with adaptability. @xcite highlighted the importance of customized content delivery in language learning, suggesting that when learners engage with material that aligns with a structured curriculum, they often experience better comprehension and retention. many researchers have ad-vocated systematically integrating curriculum content into new learning platforms to provide contextually relevant language exposure @xcite @xcite . for example, rodríguez-castro (2018) explored the potential of digital tools, such as virtual reality simulation, that map their content to official language learning curricula, ensuring that learners stay on track while taking advantage of interactive digital experiences. furthermore, @xcite ; holden and sykes (2011) demonstrated the potential of curriculum-based gamification in language learning. connecting game elements with curriculum milestones can motivate and engage learners longer. qian et al. (2023) applied lexically constrained decoding to a dialog system, encouraging it to use curriculum-aligned words and phrases, resulting in better understanding and increased interest in practicing english. our chatbot is the first to generate conversations from curricula and be trained on an open-source model. the use of pre-trained language models (plms) @xcite @xcite @xcite @xcite has enabled the generation of synthetic conversational data to enrich limited datasets, particularly in privacy-sensitive domains such as the medical domain @xcite . previous research has used plms to augment various conversational datasets @xcite @xcite @xcite . for example, @xcite and @xcite used gpt-j @xcite to generate responses tailored for emotional support dialogues and comprehension tasks, respectively. @xcite proposed a collaborative human-ai paradigm in which a human operator and gpt-3 alternate in conversation. @xcite generated dyadic and multiparty dialogues grounded on specific topic words, demonstrating outputs comparable to human-crafted ones. our approach generates in-depth conversations based on educational curricula, allowing us to shape personas, focus on specific topics, and make lexical choices during data synthesis. ## method we propose a framework for building a curriculumbased chatbot that can converse on topics derived from a given curriculum while aligning its re-sponses to the user's english proficiency level. as shown in figure 2 , our development process is divided into two parts. first, we use chatgpt to generate simulated human-to-human dialogues based on textbook topics. then, we fine-tune an open-source model to create our chatbot. ## results and discussion the full results of the user study are shown in edubot's language quality was on par with chatgpt. similar percentages of participants preferred edubot (29.2%) and chatgpt (25.0%) regarding the coherence and fluency of the chatbots' utterances. this shows that edubot produces responses of high language quality. edubot offers a diverse range of relevant dialogue topics. through topic augmentation based on the curriculum, we aim for edubot to center its conversations around topics that are relevant to but not directly listed in the textbook. significantly more participants chose edubot (50.0%) over chatgpt (16.7%) when asked which chatbot mentioned topics and content that were not directly covered in the textbook and course. this shows that edubot is capable of discussing diverse topics, compared with chatgpt, which was only prompted with topics taken directly from the textbook. at the same time, edubot's conversation content remains in line with the curriculum. when asked which chatbot's conversation topics were more related to the course, student opinions were almost evenly divided. edubot does not perform as well as chatgpt in bringing up anecdotes, examples, questions, etc., related to the course. we believe that this is because chatgpt gives longer statements that provide more material, while edubot's edubot's conversations align better with students' english proficiency levels. an equal percentage of participants (37.5%) chose edubot and chatgpt regarding which chatbot provided more vocabulary from their english course. we believe that this is because, without extra guidance, outputs produced by chatgpt are generally close to the textbook in language difficulty. this makes it difficult to highlight edubot's alignment with the students' english proficiency level. however, 37.5% of students found chatgpt used many vocabulary words they did not understand, compared to 20.8% for edubot. this shows that chatgpt's conversations were sometimes too challenging for our target users. the varied english proficiency levels among participants led to mixed results in this section. we investigate the different preferences of students with varying english levels in appendix g. edubot's conversations are more natural and realistic. participants found their conversations with edubot more natural and similar to real-life interactions. this distinction arose because, during the fine-tuning stage, edubot has access to synthetic dialogues that emulated real-life conversations of chinese college students. a higher percentage of students thought that edubot was concise and accurate (50% vs. 12.5% for chatgpt), natural and realistic (62.5% vs. 4.2% for chatgpt). on the other hand, most participants found chatgpt's responses too long and repetitive. furthermore, results show that edubot was better at guiding the conversation. 75% of students agreed that edubot asked questions to guide the conversation, compared to only 4.2% for chatgpt. using edubot, users found it easier to follow the dialogue without needing to introduce new topics to keep the conversation going. edubot acknowledges the personas of both dialogue participants. when conversing with edubot, a larger proportion of participants felt that the chatbot was aware that they were chinese college students (41.7%) compared to when they were talking to chatgpt (29.2%). edubot showed its knowledge of the user's identity by customizing its answers to the user's role. when it brought up common experiences of college students, participants could easily build upon these topics and continue the conversation. edubot was also able to showcase its assigned personas. slightly more participants felt like they were talking to two different people in their two conversations when using edubot. edubot performed better than chatgpt in providing unique perspectives (45.8% compared to 37.5%) and personal experiences (33.3% compared to 16.7%). the incorporation of diverse personas makes edobot's conversations more realistic and engaging. examples of the above two characteristics are presented and further discussed in section 6. edubot is more useful for developing english conversation skills. edubot outperformed chat-gpt in two out of three criteria for general usefulness. a greater proportion of participants preferred edubot in terms of recommending it to other students and using it to enhance their conversational skills. on the contrary, slightly more participants thought that chatgpt would be beneficial for reviewing what they had learned in class. as discussed in previous sections, the probable explanation is that chatgpt presents relevant information in more length and detail. in summary, the results show that edubot offers diverse conversation topics, takes on specific personas, acknowledges the persona of its target user, and provides natural and realistic utterances. on the other hand, the advantage of chatgpt mainly lies in its ability to provide elaborate content surrounding the conversation topic. generally, edubot is a more effective conversational partner that better cultivates the user's english conversation skills. figure 4: case 2 from edubot and case 3 from chatgpt-bot. ## case study we present several dialogue snippets from our user study to illustrate the advantages and limitations of edubot. edubot recognizes the role of the user. edubot demonstrated the ability to recognize that the user was a college student and introduced familiar, relevant topics or questions. for instance, in case 1 (figure 3 ), edubot asked about the user's university policy. by tailoring its questions to the user's background, edubot created more personalized and meaningful interactions, which are crucial for maintaining engagement and ensuring effective learning. edubot provides personal opinions and experiences. edubot formulated opinions and experiences consistent with the persona specified in the prompt, making conversations more realistic and engaging. in case 2 (figure 4 ), edubot took on the persona of a working adult and provided personal experience on continuing education after starting work. in contrast, chatgpt often did not provide realistic answers when asked about personal experiences, disrupting the natural flow of the conversation. for instance, in case 3, chatgpt struggled to offer a suitable response regarding its opinion on mooc, an online learning platform. ## conclusion and future work in this work, we present curriculum-driven edubot, a framework for developing a curriculumbased chatbot that combines the structured nature of english textbooks with the dynamic nature of chatbot interactions. we extract relevant topics from textbooks, then use llms to synthesize conversations around these topics. we fine-tune an open source model using these conversational data. our user studies show that edubot is more effective than chatgpt in facilitating curriculum-related discussions, and is also able to adjust the chatbot to match the user's english proficiency. these results demonstrate edubot's ability to provide a contextually appropriate conversational platform to develop conversation skills. in the future, there are opportunities to expand the content spectrum, incorporate multimedia elements, and introduce real-time joseph e. gonzalez, and ion stoica. 2023b. judging llm-as-a-judge with mt-bench and chatbot arena. preprint, arxiv:2306.05685.