CIMA : A Large Open Access Dialogue Dataset for Tutoring (2020)

Abstract: One-to-one tutoring is often an effective means to help students learn, and recent experiments with neural conversation systems are promising. However, large open datasets of tutoring conversations are lacking. To remedy this, we propose a novel asynchronous method for collecting tutoring dialogue via crowdworkers that is both amenable to the needs of deep learning algorithms and reflective of pedagogical concerns. In this approach, extended conversations are obtained between crowdworkers role-playing as both students and tutors. The CIMA collection, which we make publicly available, is novel in that students are exposed to overlapping grounded concepts between exercises and multiple relevant tutoring responses are collected for the same input. CIMA contains several compelling properties from an educational perspective: student role-players complete exercises in fewer turns during the course of the conversation and tutor players adopt strategies that conform with some educational conversational norms, such as providing hints versus asking questions in appropriate contexts. The dataset enables a model to be trained to generate the next tutoring utterance in a conversation, conditioned on a provided action strategy.

## introduction there is a pressing societal need to help students of all ages learn new subjects. one-on-one tutoring is one of the most effective techniques for producing learning gains, and many studies support the efficacy of conversational tutors as educational aids @xcite @xcite . tutoring dialogues should exhibit a number of important properties that are not present in exist-ing open datasets. the conversation should be grounded around common concepts that both the student and the tutor recognize are the topics to be learned @xcite . the conversation should be extended, that is, long enough for the student to be exposed to new concepts, givin students the opportunity to recall them in future interactions. the collection should contain varied responses, to reflect the fact that there is more than one valid way for a tutor to respond to a student at any given point in the conversation. and lastly, the dialogue should not contain personally identifiable information so it can be available as open access data. we propose a novel method for creating a tutoring dialogue collection that exhibits many of the properties needed for training a conversational tutor. in this approach, extended conversations are obtained between crowdworkers role-playing as both students and tutors. students work through an exercise, which involves translating a phrase from english to italian. the workers do not converse directly, but rather are served utterances from prior rounds of interaction asynchronously in order to obtain multiple tutoring responses for the same conversational input. special aspects of the approach are: ## tutoring dataset creation we create two dialogue datasets within cima: shapes and prepositional phrases with colored objects. ## dataset statistics an analysis of the conversations found that the data contains several interesting properties from an educational perspective. this section summarizes overall statistics of the data collected; the subsequent two sections summarize phenomena associated with the student and tutor data. ## student interactions by examining a student's interactions over the course of a 20-turn hit, we find that students take fewer turns on average to complete an exercise at the end than at the beginning of a hit. we examine the number of turns students take before reaching the correct answer, as we hypothesize this will decrease as students have more exposure to domain concepts. we note this could be due to many factors, such as the students becoming more comfortable with the task or system or learning italian phrases they were exposed to in prior questions. we measure this with the prepositional phrase domain, because the students interacted with the system for 20 turns, compared to the 5-turn interactions with the shape task. for a given hit, we compare the number of student turns needed to produce their first correct answer with how many turns were needed for their final correct answer. 3 for each student, we calculate the difference between the number of turns required between their first and final correct answers. the average difference is -0.723, indicating students required fewer turns to achieve their last correct answer than their first. thus the data set might contain evidence of learning, although it could be as simple as student workers learning how to more efficiently ask questions of the system. ## tutor phenomena we examine several characteristics about the tutor interactions: (i) properties about the language tutors use in their responses, (ii) how tutors respond to different student action types, and (iii) characterizing if and how tutors agree when presented with identical student input. ## tutoring model we claim cima is useful to train neural models for tutoring tasks. to explore this, we train a generation model (gm) aimed at producing a tutoring response conditioned on two past conversation utterances. 5 an example input would be: hint, correction, e di fronte al, giallo, coniglio, is in front of the, yellow, bunny, <@xmath0: well, "bunny" is "coniglio" student: il gatto e di fronte al coniglio. in this representation, domain information and an intended set of actions to take is separated with a special @xmath1. model training details are in appendix e. we split the data along conversations into 2296 train, 217 development, and 107 test utterances. ## discussion our analysis finds that tutors are more unified in an action strategy when a student asks a question than other actions. this is consistent with the findings that (i) when tutors agree, they are more likely to agree on a hint and (ii) the most likely action in response to a student question is a hint. overall tutor agreement was low among the dataset (18.1%), indicating the potential capture of divergent tutoring strategies. future work can leverage this disagreement to explore the multiple potential actions to take when responding to a student. our preliminary experiments show cima can be used to train a model that can generate text conditioned on a desired actions. future work should explore more complex models utilizing cima, as well as exploring the other unique qualities of the collection, such as the shared image representation, multiple tutoring utterances for each conversation, and link between the two domains. tutoring responses marked as not making sense should be explored, to both improve the process of serving student responses as well as correcting a model when a generated response veers the conversation off track. a benefit to having this explicitly logged is that the collection contains labeled negative examples of tutoring responses, which can be leveraged in training models. ## limitations while past work utilized crowdworkers to collect tutoring utterances @xcite and for peer learning studies @xcite , future work should examine the similarities and differences between the language and actions taken by crowdworkers and actual tutors and students engaged in the learning process. because we were working with untrained crowdworkers, we were constrained in the complexity of language learning concepts we could include in cima. it is possible that the resulting dataset only transfers to novice language learners. future work should examine how well this generalizes to a real language learning setting and how general tutoring language and strategies that emerge from our domain transfer to more complex ones. the dataset currently does not distinguish the type of hint or correction tutors make. examples include providing direct corrections versus indirect feedback which states the error and allows the student to self-correct @xcite . future work on cima can examine the prevalence of these different types of feedback and potential benefits or shortcomings. ## conclusion we present cima: a data collection method and resulting collection of tutoring dialogues which captures student interactions and multiple accompanying tutoring responses. two datasets of differing complexity have direct applicability to building an automatic tutor to assist foreign language learning, as we examine with a preliminary model. cima has the potential to train personalized dialogue agents which incorporate longer-term information, have a well-defined goal to have a student learn and recall concepts, and can explore different correct utterances and actions at given times.