Opportunities and Challenges in Neural Dialog Tutoring (2023)

Abstract: Designing dialog tutors has been challenging as it involves modeling the diverse and complex pedagogical strategies employed by human tutors. Although there have been significant recent advances in neural conversational systems using large language models and growth in available dialog corpora, dialog tutoring has largely remained unaffected by these advances. In this paper, we rigorously analyze various generative language models on two dialog tutoring datasets for language learning using automatic and human evaluations to understand the new opportunities brought by these advances as well as the challenges we must overcome to build models that would be usable in real educational settings. We find that although current approaches can model tutoring in constrained learning scenarios when the number of concepts to be taught and possible teacher strategies are small, they perform poorly in less constrained scenarios. Our human quality evaluation shows that both models and ground-truth annotations exhibit low performance in terms of equitable tutoring, which measures learning opportunities for students and how engaging the dialog is. To understand the behavior of our models in a real tutoring setting, we conduct a user study using expert annotators and find a significantly large number of model reasoning errors in 45% of conversations. Finally, we connect our findings to outline future work.

## introduction the goal of dialog tutoring research is to build systems that can tutor students using natural language conversation @xcite . for several decades, learning scientists have been studying the * equal contribution. features of domain-specific dialog tutoring systems that engender learning in students @xcite @xcite @xcite and have established strong learning gains that are even comparable to human tutoring in specific domains @xcite . however, these systems require extensive authoring of materials by teachers @xcite and therefore cannot fully utilize the scalability of online learning. building dialog tutors is technically challenging as tutoring dialogs typically exhibit properties that are absent in other forms of dialog. tutoring dialogs are often long, enabling students to be exposed to the concepts in a way that they can use them in future @xcite , and grounded in the learning scenarios @xcite . finally, good dialog tutors are engaging and create opportunities to learn, providing students space to seek and provide explanations, and self-reflect @xcite . the growing success of deep neural network based language generators in other dialog settings @xcite suggests new possibilities in dialog tutoring that could scale beyond domain-specific approaches. however, despite their promise, advances in neural generative models have seen little adoption in dialog tutoring. in this paper, we contribute a comprehensive study of the applicability of neural generative models in tutoring. we formally introduce the dialog tutoring task and analyze existing tutoring datasets ( §2). then, we describe several generative and retrieval-based models for dialog tutoring ( §3) and benchmark them on two open-access dialog tutoring datasets for language learning: cima @xcite , a crowdsourced role-played dataset for learning prepositional phrases in italian) and teacher-student chatroom corpus (tscc) @xcite , a one-to-one english tutoring dataset from an online chatroom) ( §5.1). we evaluate our models on various automatic metrics ( §4.2) as well as two human evaluation studies: an evaluation of the quality of the generated response with respect to various measures of goodness ( §6.1), as well as a more realistic user study with a learning interface ( §6.2). overall, while we find that pretrained models improve over simpler baselines in terms of automatic metrics, our consequent human evaluation reveals several shortcomings that ought to be addressed before these models can be adopted in the real world. we find that while neural generative models can model more constrained learning settings well, they struggle when the learning goal is more open-ended. specifically, these models are unable to understand and reason about student solutions and misconceptions, and thus, are unable to use effective pedagogical strategies. we find that the field of dialog tutoring is significantly limited by the quantity and quality of available datasets. the available datasets are both too small and not rich enough to capture the nuances of the dialog tutoring problem. our analysis also reveals the inadequacy of automatic evaluation metrics for capturing tutoring quality. not only are the existing metrics unable to capture faithfulness to the learning material and the student dialog history, but they also cannot capture moves of good human tutors that allow learners the space for reflection, explanation, follow-ups, and real engagement in the process of learning. based on our findings, we end with an outline of potential avenues of future research ( §7). we hope that our paper will bring attention to this underexplored natural language processing application with the potential for significant social good. @xcite the dialog tutoring task dialog tutoring can be described as a multi-turn interaction between two interlocutors, where one performs the role of a teacher seeking to teach the other interlocutor who acts in the role of a student. we then can describe a dialog tutoring session formally as a sequence of turns @xmath0, . . . , u |h| ) that are taken by either of the interlocutors. each turn u @xmath1. further, each turn u t can be associated with a sequence of dialog acts a @xmath2. the dialog act is a key aspect of dialog tutoring as it can refer to the teaching strategy employed by the tutor. these may include strategies such as providing a hint or seeking a clarification (see appendix a for more details). the set of dialog acts a is usually fixed according to a predefined taxonomy and may be split into two subsets @xmath3, each corresponding to the teacher and student role. each dialog session h may also be accompanied with some grounding information k, which grounds the response in relevant information and may refer to the teaching material that needs to be taught to the student. this information k may come in various formats, including images and videos. however, we restrict ourselves to using only text-based grounding in this work such that @xmath4.g., a textbook with a set of chapters). in section 3 we derive different methods to model the role of the teacher, to which we restrict this work. ## dialog tutoring models after introducing the dialog tutoring task, this section highlights the models we evaluate on the task. we note that our aim is an analysis of existing models. we explore turn-level models that can generate a teacher response y := u t+1 given a tutoring session @xmath5, . . . , u |h| ). during training, we obtain the dialog history by teacher forcing, i.e., we take the ground-truth dialog history. furthermore, we do not model the problem of retrieving grounding information but rather assume it as given. ## results in this section, we summarize our main findings in terms of automatic evaluation. first, we give an overview of the performance of different models that we train on cima and tscc in section 5.1. then, we assess their ability to stay faithful to teaching strategies (section 5.2) and study how grounding annotations can influence the faithfulness of neural dialog tutors (section 5.3), before studying their scaling behavior with dataset size and complexity (section 5.4) and their generalization capabilities (section 5.5). we then finish with an assessment of using education-specific data for pretraining (section 5.6). ## human evaluation we further evaluate previously assessed models with human judgments firstly by obtaining quality estimates according to different criteria and secondly by conducting a simulation study, where expert annotators are asked to provide novel rewritings of existing conversations and to categorize errors made by the model. ## discussion: towards more equitable and faithful tutoring systems in this section, we outline directions of research that we think can be important steps towards more equitable and faithful tutoring models. namely, we first address the small scale and quality of current tutoring datasets and cast doubt on the crowdsourcing data quality checks. then, we suggest ways of improving the underperformance of both equitable tutoring and teaching strategy prediction identified in current generative models under these constraints by drawing from learning sciences literature. finally, we outline desiderata for more reliable dialog evaluation of neural tutoring models. datasets based on the analysis in §2.1 and models so far, dialog tutoring models have only covered limited domain-specific settings linked to a particular activity, such as learning italian prepositions or solving math word problems. we argue that the community could benefit from working on problems common to learning in general, for example tracking problem-solving states and modeling pedagogies used by teachers. here, knowledge tracing (corbett and anderson, 1994) (the problem of estimating students' skill mastery level) could be used for tracking problem-solving states and increasing the coherence of dialog tutoring conversations and dialog act selection performance which would contribute to better modeling of global teaching strategies. furthermore, validated instruction quality coding schemes @xcite used by classroom teachers can be computationally modeled @xcite and incorporated into models. we also think that recently proposed constrained decoding approaches that can balance between multiple criteria @xcite hold great promise in improving faithfulness in complex tutoring dialogs. finally, as data collection is labor-intensive in expert domains, we see great potential in few-shot learning methods, such as prompt-based methods @xcite . evaluation our experiments highlight the insufficiencies of current automatic dialog evaluation metrics, as both bleu and bertscore show comparatively low correlation with our collected human judgements from §6.1. this is in line with previous research @xcite ) and shows the necessity not only for better automatic evaluation metrics but also for verification based on human judgements or user studies that should incorporate criteria relevant to tutoring (e.g., equitable tutoring outcomes). metrics that incorporate task success, which have been used in task-oriented dialog systems @xcite , are a promising direction of future research for automatic evaluation. ## conclusion in this work, we reflected on the state of research in dialog tutoring and explored the potential of neural generative models in this domain. we found some promising initial results with these models in comparison to rule-or retrieval-based methods. however, we also established limitations of currently available benchmarks and evaluation criteria. furthermore, we showed that there are a number of challenges that need to be addressed before neural generative models of text can be deployed as intelligent tutoring systems on a larger scale, such as controllability and being able to model a sound pedagogical strategy. based on these findings, we outline potential avenues for future research.