Strategize Before Teaching: A Conversational Tutoring System with Pedagogy Self-Distillation (2023)

Abstract: Conversational tutoring systems (CTSs) aim to help students master educational material with natural language interaction in the form of a dialog. CTSs have become a key pillar in educational data mining research. A key challenge in CTSs is to engage the student in the conversation while exposing them to a diverse set of teaching strategies, akin to a human teacher, thereby, helping them learn in the process. Different from previous work that generates responses given the strategies as input, we propose to jointly predict teaching strategies and generate tutor responses accordingly, which fits a more realistic application scenario. We benchmark several competitive models on three dialog tutoring datasets and propose a unified framework that combines teaching response generation and pedagogical strategy prediction, where a self-distillation mechanism is adopted to guide the teaching strategy learning and facilitate tutor response generation. Our experiments and analyses shed light on how teaching strategies affect dialog tutoring.

## introduction decades of research effort @xcite @xcite has been put into building intelligent tutoring systems (itss). an important feature of these systems is the ability to customize the instructional activities and strategies based on the learner's characteristics and needs @xcite . conversational tutoring systems (ctss) that aim to offer automated tutoring through natural language dialog is a key pillar of its research. earlier work in ctss was based on conventional techniques such as bayesian techniques with rule engines @xcite and hybrid neural networks @xcite . while various advanced neural approaches have been applied to open-domain @xcite @xcite and task- teaching strategy tutor response restating let me say back what i heard. pressing for accuracy can you tell us the steps you used to find the answer? (a) two examples of teaching strategy and tutor response tutor: ok, now we have 'get cut off', 'put someone through' and 'get through' scaffolding tutor: i was talking to her, but suddenly we ___ ___ ___ (i couldn't hear her anymore) can you choose one? eliciting student: got cut off oriented dialogue systems @xcite @xcite , conversational tutoring systems have not benefited from the development of these technologies @xcite . human teachers use a number of nuanced teaching strategies in the classroom during interactions with students; these strategies are tailored to keep the students engaged in the conversation and learn knowledge efficiently. we show some examples of teaching strategies and interactions between the tutor and the student in fig. 1 . previous work has attempted to model these teaching strategies in different ways -e.g., @xcite contributed a teaching strategy classification model and @xcite proposed a response generation model based on given teaching strategies of next response. in this work, we benchmark several neural dialog models on three conversational tutoring datasets, cima @xcite , tscc @xcite and talkmoves @xcite , and contribute a unified framework based on pretrained language models, where teaching strat-egy prediction and response generation are jointly trained. as predicting a teaching strategy merely by the historical context is more difficult than when we are also given the target tutor response, we also propose a pedagogy distillation mechanism that allows teaching strategy prediction to learn from the soft labels which are produced by the prediction with target response. the soft labels learned from the target response provides the model knowledge about various interrelationships between teaching strategies that hard labels lack. this approach is believed to be able to alleviate the learning difficulty @xcite , which is particularly important, especially when the data imbalance and scarcity issues are severe -often the case in conversational tutoring data. in summary, we are the first to benchmark 1 several competitive models for conversation tutoring system on all three datasets that are currently available. besides, we propose a unified framework that can predict teaching strategy and generate tutoring responses accordingly, which is enhanced by a selfdistillation mechanism. our experiments validate the positive effects of teaching strategy to guide generation and the importance of predicting strategy first and then generate response accordingly. ## related work a classical intelligent tutoring system generally has three modules @xcite : (i) expert module that includes the knowledge that the student wants to learn (carter, 2014)). (ii) student module that can adjust the level of student (e.g., primary/middle school, non-native/native speaker), student's knowledge deficiency, etc. (iii) pedagogical module that focuses on the strategies of teaching. in expert module, the knowledge is usually domain specific, such as computer programming @xcite , mathematics @xcite , italian @xcite , english @xcite . many technologies have been used in the expert module, such as bayesian techniques with rule engines @xcite and hybrid neural networks @xcite . for pedagogical module, to our best knowledge, there are only three publicly available datasets that provide the pedagogy information. they are cima (stasaski 1 the code can be found in https://github.com/ lingzhi-wang/tutorsystem source sequence encoder ## experimental setup datasets. we use three datasets to do the experiments. they are cima @xcite , tscc @xcite and talkmoves @xcite . cima contains one-to-one conversations that focus on teaching students to translate a phrase from english to italian. tscc focuses on teaching english for eight non-native englishspeaking students. talkmoves is constructed by transcripts of math classrooms. parameter setting. our implementation is based on fairseq @xcite . we split the data into 8:1:1 for training, validation and test. all the hyperparameters are chosen by grid-search based on the validation performance. we use bart-base foot_0 and mbart-large foot_1 models to initialize our model, respectively. bart-base model has 6 layers of encoder and decoder with 768 hidden dimension, while mbart-large has 12 layers of encoder and decoder with 1024 hidden dimension. the parameter sizes for the two models initialized with bart and mbart are 199m and 816m, respectively. we use one nvidia rtx 3090 gpu to train our model. during training, we set the max tokens of each batch to 1024 (for bart, or 512 for mbart) with an update frequency of 4. we adopt adam optimizer (kingma and ba, 2015) with learning rate selected in {1e-4, 5e-5, 2e-5, 1e-5} and warm-up updates selected in {200, 500, 1000} followed by a polynomial decay scheduler. dropout strategy @xcite with dropout rate selected in {0.2, 0.4} and l 2 regularization with 0.01 effect value, as well as early stoping based on validation performance, are used to alleviate overfitting. we set the tradeoff values among the losses as λ = 1.0, γ = 1.0 and δ = 0.2. during inference, predicting threshold θ = 0.3 and beam size is set to 5. ## conclusion in this work, we benchmarked neural models on various conversational tutoring datasets and proposed a self-distillation based model that jointly trains a teaching strategy prediction model and a response generation model. experiments on three conversational tutoring datasets show that our model outperforms various standard baselines by a significant margin. finally, we ended with an interesting case study to demonstrate the importance of strategizing before teaching.