Communicative Grounding of Analogical Explanations in Dialogue: A Corpus Study of Conversational Management Acts and Statistical Sequence Models for Tutoring through Analogy (2021)

Abstract: We present a conversational management act (CMA) annotation schema for one-to-one tutorial dialogue sessions where a tutor uses an analogy to teach a student a concept. CMAs are more fine-grained sub-utterance acts compared to traditional dialogue act mark-up. The schema achieves an inter-annotator agreement (IAA) Cohen Kappa score of at least 0.66 across all 10 classes. We annotate a corpus of analogical episodes with the schema and develop statistical sequence models from the corpus which predict tutor content related decisions, in terms of the selection of the analogical component (AC) and tutor conversational management act (TCMA) to deploy at the current utterance, given the student’s behaviour. CRF sequence classifiers perform well on AC selection and robustly on TCMA selection, achieving respective accuracies of 61.9% and 56.3% on a cross-validation experiment over the corpus.

## introduction the motivation for our work is two-fold; firstly it derives from our interest to investigate which conversational management acts (cmas) are prevalent in tutorial explanations which use analogies as a pedagogical strategy, and secondly, our interest to determine what the optimal computational model is for selecting the appropriate analogical components (acs) and tutor conversational management acts (tcmas) for an artificial tutoring agent. these findings represent a milestone towards our end goal of building a conversational ai system which is capable of tutoring in introductory computer science topics on a speech based modality. the paper is presented in the following sections: in section 2 we present theoretical and empirical foundations on dialogue acts (das), tutorial dialogue, hidden markov models (hmms) and conditional random fields (crf) modelling; section 3 presents the first contribution of this paper, which is the development of an annotation schema of tutor conversational management acts (tcmas) and student conversational management acts (scmas); section 4 presents a corpus study on dialogues annotated using the developed schema; sections 5 and 6 present the second contribution of this investigation, (i) predictive models on analogical component (ac) uttered by the tutor and (ii) models predicting tcmas, followed by concluding the findings and future work in section 7. ## annotation schema to develop a schema of tutor conversational management acts (tcma) and student conversational management acts (scma), we selected 600 utterances from analogical episodes tutor so, the way we can think of a stack is, kind of like a brick wall, right? tutor there is uh, we lay a brick down, and every time we put something else on the stack we always lay on top of the previous one. tutor right? student yeah. tutor when we build a wall we build the bottom up. student yeah, thats true. ## corpus study the cstd corpus @xcite contains 54 sessions of annotated tutorial dialogues. a sub-corpus composed of the 138 sequences of utterances marked as analogical episodes was tutor conversational management act description example question response (qr) the tutor responds to a question asked by the student. student: "and these two lists are totally separate?" tutor: "yeah, these are totally separate." elicit response (er) questions by the tutor to test if the student understands what he or she is talking about, or to see if the student is paying attention. "it's stored +// it stores information, okay?" "right?" diagnosing base (db) the tutor asks a question to check if the student is familiar with the base concept of an analogy. "are you familiar with legos?" floor initiating or floor maintaining discourse marker (fim) floor initiating in one utterance. floor maintaining (the expression is located in the middle of the utterance) "uh are you familiar with legos?" "um what the stack is # is a way to hold objects such as uh integers or numbers, letters, words." "okay, a binary search is like a family tree" "now if you notice, there is no. . . " "so now if i'm go again. . . " "so we could push..." acknowledge backchannel (abk) appears immediately after the student issues a "backchannel" student: right. [bk] tutor: "right? [abk] the student issues a backchannel. "uh huh", "yeah", "mmhm", "right", "ok","okay" backchannel through repetition (br) the student backchannels through repetition. tutor: . . . the head is for the next line. student: line ok. ## predicting a tutor's selection of analogical component using sequence modelling given the eventual goal of building an artificial tutor which can ground analogies with a tutee over time, we use the corpus described above to train two types of sequence model: (i) the tutor's sequential decisions of selecting an appropriate analogical component (ac) and (ii) the tutor selection of the appropriate conversational management act (tcma) for their next utterance. we test both a generative sequence model, a hidden markov model (hmm) classifier, and a discriminative model, a conditional random field (crf) classifier. we use markovian sequence models due to the fact there is a regularity in analogical structure following the semantic wave @xcite . due to the relatively small size of the possible training data (138 sequences, 3039 tutor utterances), we do not use neural net based sequence architectures in this work. the first type of decision we model is predicting the ac to be communicated in the next tutor utterance from the 3 classes of a base component (b), target (t) or both (bt). our ac prediction models all take as input a sequence of elements which represent the student's last utterance after the most recent tutor utterance (so this could be no utterance (∅) if the tutor continues to hold the floor). we are in effect modelling the decision a dialogue manager would take in a dialogue system to select the analogical component to communicate in the next utterance. we set up an hmm and crf to do this prediction as explained in the following sections. ## predicting tutor conversational management acts the second decision process we model, given student scmas and an ac selected by a model such as that just described (though here using the gold standard ac tags), is the selection of one of the 12 either single or compound tcmas observed in the corpus shown in