Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions (2021)

Abstract: In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers’ uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers’ uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (pJSD), estimated via next utterance classification; (3) conducting a linguistically-motivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, pJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers.

## introduction building on the interlocutor's contribution via, for example, acknowledgment, repetition or elaboration (figure 1 ), is known as uptake and is key to a successful conversation. uptake makes an interlocutor feel heard and fosters a collaborative interaction @xcite ), 1 code and annotated data: https://github.com/ ddemszky/conversational-uptake i added 30 to 70… okay. good, you did the first step. which is especially important in contexts like education. teachers' uptake of student ideas promotes dialogic instruction by amplifying student voices and giving them agency in the learning process, unlike monologic instruction where teachers lecture at students @xcite @xcite . despite extensive research showing the positive impact of uptake on student learning and achievement @xcite @xcite , measuring and improving teachers' uptake at scale is challenging as existing methods require manual annotation by experts and are prohibitively resource-intensive. we introduce a framework for computationally measuring uptake. first, we create and release a dataset of 2246 student-teacher exchanges extracted from us elementary math classroom transcripts, each annotated by three domain experts for teachers' uptake of student contributions. we take an unsupervised approach to measure uptake in order to encourage domain-transferability and account for the fact that large amounts of labeled data are not possible in many contexts due to data privacy reasons and/or limited resources. we conduct a careful analysis of the role of repetition in uptake by measuring utterance overlap and similarity. we find that the proportion of student words repeated by the teacher (%-in-t) captures a large part of uptake, and that surprisingly, wordlevel similarity measures consistently outperform sentence-level similarity measures, including ones involving sophisticated neural models. to capture uptake phenomena beyond repetition and in particular those relevant to teaching (e.g. question answering), we formalize uptake as a measure of the reply's dependence on the source utterance. we quantify dependence via pointwise jensen-shannon divergence (pjsd), which captures how easily someone (e.g., a student) can distinguish the true reply from randomly sampled replies. we show that pjsd can be estimated via cross-entropy loss obtained from next utterance classification @xcite . we train a model by fine-tuning bert-base @xcite via nuc on a large, combined dataset of student-teacher interactions and switchboard @xcite . we show that scores obtained from this model significantly outperform our baseline measures. using dialog act annotations on switchboard, we demonstrate that pjsd is indeed better at capturing phenomena such as reformulation, question answering and collaborative completion than %-in-t, our best-performing baseline. our manual analysis also shows qualitative differences between the models: the examples where pjsd outperforms %-in-t are enriched by teacher prompts for elaboration, an exemplar for dialogic instruction @xcite . finally, we find that our pjsd measure shows a significant linear correlation with outcomes such as student satisfaction and instruction quality across three different datasets of student-teacher interactions: the ncte dataset @xcite , a one-on-one online tutoring dataset, and the simteacher dataset @xcite . these results provide evidence for the generalizability of our pjsd measure and for its potential to serve as an automated tool to give feedback to teachers. ## background on uptake uptake has several linguistic and social functions. (1) it creates coherence between two utterances, helping structure the discourse @xcite @xcite . (2) it is a mechanism for grounding, i.e. demonstrat-ing understanding of the interlocutor's contribution by accepting it as part of the common ground (shared set of beliefs among interlocutors) @xcite . (3) it promotes collaboration with the interlocutor by sharing the floor with them and indicating what they have said is important @xcite . there are multiple linguistic strategies for uptake, such as acknowledgment, collaborative completion, repetition, and question answering -see figure 1 for a non-exhaustive list. a speaker can use multiple strategies at the same time, for example, t 3 in figure 1 includes both acknowledgment and repetition. different strategies can represent lower or higher uptake depending on how effectively they achieve the aforementioned functions of uptake. for example, @xcite argues that repetition is a highly pervasive and effective strategy for ratifying listenership and building a coherent discourse. in education, high uptake has been defined as cases where the teacher follows up on the student's contribution via a question or elaboration @xcite . we build on this literature from discourse analysis and education to build our dataset, to develop our uptake measure and to compare the ability of different measures to capture key uptake strategies. ## a new educational uptake dataset despite the substantial literature on the functions of uptake, we are not aware of a publicly available dataset labeled for this phenomenon. to address this, we recruit domain experts (math teachers and raters trained in classroom observation) to annotate a dataset of exchanges between students and teachers. the exchanges are sampled from transcripts of 45-60 minute long 4th and 5th grade elementary math classroom observations collected by the national center for teacher effectiveness (ncte) between 2010-2013 @xcite . the transcripts represent data from 317 teachers across 4 school districts in new england that serve largely low-income, historically marginalized students. transcripts are fully anonymized: student and teacher names are replaced with terms like "student", "teacher" or "mrs. h". preparing utterance pairs. we prepare a dataset of utterance pairs (s, t ), where s is a student utterance and t is a subsequent teacher utterance. the concept of uptake presupposes that there is something to be taken up; in our case that the student utterance has substance. for example, short student utterances like "yes" or "one-third" do not present many opportunities for uptake. based on our pilot annotations, these utterances are difficult for even expert annotators to label. therefore, we only keep utterance pairs where s contains at least 5 tokens, excluding punctuation. we also remove all utterance pairs where the utterances contain an [inaudible] marker, indicating low audio quality. out of the remaining 55k (s, t ) pairs, we sample 2246 for annotation. annotation. given that uptake is a subjective and heterogeneous construct, we relied heavily on domain-expertise and took several other quality assurance steps for the annotation. as a result, the annotation took six months to develop and complete, longer than most other annotations in nlp for a similar data size (∼2k examples). our annotation framework for uptake is designed by experts in math quality instruction, including our collaborators, math teachers and raters for the mathematical quality instruction (mqi) coding instrument, used to assess math instruction (teaching project, 2011). in the annotation interface, raters can see (1) the utterance pair (s, t ), (2) the lesson topic, which is manually labeled as part of the original dataset, and (3) two utterances immediately preceding (s, t ) for context. annotators are asked to first check whether (s, t ) relates to math -e.g. "can i go to the bathroom?" is unrelated to math. if both s and t relate to math, raters are asked to select among three labels: "low", "mid" and "high", indicating the degree to which a teacher demonstrates that they are following what the student is saying or trying to say. the annotation framework is included in appendix a. we recruited expert raters (with experience in teaching and classroom observation) whose demographics were representative of us k-12 teacher population. we followed standard practices in education for rater training and calibration. we conducted several pilot annotation rounds (5+ rounds 3 to enable potential analyses on the temporal dynamics of uptake, we randomly sampled 15 transcripts where we annotate all (s, t ) pairs (constituting 29% of our annotations). the rest of the pairs are sampled from the remaining data. with a subset of raters, 2 rounds involving all 13 raters), quizzes for raters, thorough documentation with examples, and meetings with all raters. after training raters, we randomly assign each example to three raters. post-processing and rater agreement. ## uptake as overlap & similarity as we see in ## downstream application to test the generalizability of our uptake measures and their link to instruction quality, we correlate pjsd and %-in-t with educational outcomes on three different datasets of student-teacher interactions ( ## related work prior computational work on classroom discourse has employed supervised, feature-based classifiers to detect teachers' discourse moves relevant to student learning, such as authentic questions, elaborated feedback and uptake, treating these moves as binary variables @xcite @xcite @xcite . our labeled dataset, unsupervised approach (involving a state-of-the art pre-trained model), and careful analysis across domains are novel contributions that will enable a fine-grained and domain-adaptable measure of uptake that can support researchers and teachers. our work aligns closely with research on the computational study of conversations. for example, measures have been developed to study constructiveness @xcite , politeness @xcite and persuasion @xcite in conversations. perhaps most similar to our work, zhang and danescu-niculescu-mizil (2020) develop an unsupervised method to identify therapists' backwardand forward-looking utterances, with which they guide their conversations. we also draw on work measuring discourse coherence via embedding cosines @xcite , or via utterance classification @xcite , the latter of which is used also for building and evaluating dialog systems @xcite . our work extends these two families of methods to human conversation and highlights the different linguistic phenomena they capture. finally, our work shows the key role of coherence in the socially important task of studying uptake. ## conclusion we propose a framework for measuring uptake, a core conversational phenomenon with particularly high relevance in teaching contexts. we release an annotated dataset and develop and compare unsupervised measures of uptake, demonstrating significant correlation with educational outcomes across three datasets. this lays the groundwork (1) for scaling up teachers' professional development on uptake thereby enabling improvements to education, (2) for conducting analyses on uptake across domains and languages where labeled data does not exist and (3) for studying the effect of uptake on a wider range of socially relevant outcomes.