Computationally Identifying Funneling and Focusing Questions in Classroom Discourse (2022)

Abstract: Responsive teaching is a highly effective strategy that promotes student learning. In math classrooms, teachers might funnel students towards a normative answer or focus students to reflect on their own thinking depending their understanding of math concepts. When teachers focus, they treat students’ contributions as resources for collective sensemaking, and thereby significantly improve students’ achievement and confidence in mathematics. We propose the task of computationally detecting funneling and focusing questions in classroom discourse. We do so by creating and releasing an annotated dataset of 2,348 teacher utterances labeled for funneling and focusing questions, or neither. We introduce supervised and unsupervised approaches to differentiating these questions. Our best model, a supervised RoBERTa model fine-tuned on our dataset, has a strong linear correlation of .76 with human expert labels and with positive educational outcomes, including math instruction quality and student achievement, showing the model’s potential for use in automated teacher feedback tools. Our unsupervised measures show significant but weaker correlations with human labels and outcomes, and they highlight interesting linguistic patterns of funneling and focusing questions. The high performance of the supervised measure indicates its promise for supporting teachers in their instruction.

## introduction students are more engaged and learn more when teachers pose carefully chosen questions to draw out student thinking, and then attend closely to what students say @xcite . one way that teachers do this is by using focusing question patterns; i.e., "attending to what the students are thinking, pressing them to communicate their thoughts figure 1 : example teacher utterance and possible student replies, illustrating the difference in funneling (top exchange) and focusing (bottom exchange). @xcite clearly, and expecting them to reflect on their thoughts and those of their classmates" (national council of teachers of mathematics, 2014, hereafter nctm). focusing is often contrasted to the less effective yet more common question pattern of funneling, where teachers pose "a set of questions to lead students to a desired procedure or conclusion, while giving limited attention to student responses that veer from the desired path" @xcite . the use of focusing questioning patterns has been linked to better student learning outcomes and confidence in mathematics @xcite . supporting teachers to use more focusing question patterns requires first helping them to identify the extent to which they are focusing or funneling in their own classrooms. however, the current methods of measuring funneling and focusing are resource intensive, requiring manual classroom observation (e.g., @xcite . developing computational methods for identifying funneling and focusing thus present an opportunity to provide automated feedback for questioning patterns at scale. recent tools that provide automated feedback to teachers on discourse moves have been effective at improving their uptake of student contributions and student outcomes @xcite and helped raise awareness about different instructional talk moves @xcite . one promising application of an automated measure of questioning patterns is to build a similar tool that encourages teachers to engage with their students by asking more focusing questions. we propose several approaches for computationally identifying funneling and focusing questions, including supervised and unsupervised modeling. in order to develop our approaches, we create a dataset of 2,348 student-teacher exchanges sampled from elementary math classroom transcripts, each annotated by three domain experts for teachers' use of funneling and focusing questions, or neither. then, we fine-tune a supervised roberta model @xcite on the annotated data. this model has the highest correlation of .76 with human judgments, among our proposed models. we also explore several unsupervised learning approaches, in order to encourage domaintransferability, to account for the lack of labeled data in most educational settings, and to analyze the linguistic patterns that drive funneling vs focusing questions. our first unsupervised model hinges on the assumption that the range of possible student responses are narrower for funneling questions than for focusing ones. in figure 1 , we see that the teacher's funneling questions about the rise and run are quantitative in nature, so we can more confidently predict that the student response will be a number. conversely, focusing utterances, which encourages students to reflect on their own thinking, tend to have a wider range of valid responses. the teacher's focusing question in figure 1 shows that the students can think about the slope in many different ways, so we can less confidently predict what the student reply will be. following this intuition, we adapt zhang and danescu-niculescu-mizil (2020)'s measure of forwards-range, an unsupervised measure that quantifies the strength of our expectation of a reply to a given utterance. we also use other linguistic features informed by educational theory as measures to identify funneling and focusing. since focusing examples probe student thinking and understanding, we use the count of cognitive verbs present in an utterance as one of the features. question words and phrases also provide insight into classifying closed-ended and open-ended questions, so we include both the count of unigram and bigram question words as features. to further validate our measures and to understand the link between funneling and focusing and educational outcomes, we correlate our measures with observation scores of instruction quality and student engagement and with value-added scores. value-added scores are statistical estimates of a teacher's contribution to student test score growth, which serve important indicator of student learning and achievement. we find that our roberta model correlates strongly with all of these outcomes, which is a significant finding in the context of educational measurement (kraft, 2020), and it indicates the promise of this measure to support teachers and students. ## contributions we make the following contributions in this paper. ## related work many researchers have measured the types of question patterns that teachers use in classrooms by hiring and training raters to manually code transcripts of teacher-student discourse @xcite @xcite . while this measurement approach has been useful for identifying effective teaching practice in well-funded large-scale research studies, it is too costly to be scalable. computational methods for measuring question patterns in classrooms offer both the potential to undertake more research in this area, as well as the potential to support teachers to improve their classroom practice by automatically coding aspects of their classroom discourse for them to review. prior work in computationally analyzing classroom discourse has employed a variety of techniques to automatically detect teacher discourse variables. recent advances in natural language processing has led to a larger presence of work applying neural methods with varying levels of success in detecting classroom discourse variables, such as semantic content, instructional talk, and elaborated evaluation @xcite . for unsupervised approaches, demszky et al. (2021a), which is also most similar to our work in terms of approach and dataset, propose an unsupervised measure of teachers' uptake of students' contributions, and we use their sample in our annotation for funneling and focusing. other computational work on questions in classroom discourse has focused on detecting questions in live classroom audio @xcite and measuring the authenticity of questions in classroom discourse @xcite . our task closely relates to the task of detecting authentic questions but instead of using the class framework used by prior work, we draw on the math education literature to develop our own coding instrument for funneling and focusing. in addition, while prior work in computationally analyzing questions uses feature-based classification, we also apply state-of-the-art neural machine learning models to solve this task. our proposed task of identifying funneling and focusing questions is situated among related dialogue tasks where the goal is to predict a label for a set of turns in dialogue. general approaches to this task have employed supervised classifiers in a variety of settings, such as to classify sarcasm in social media dialogue and participant roles in cyberbullying @xcite . similar to our approach of identifying patterns that generalize beyond annotated data, others in this domain have also found meaningful patterns and features in labeled data that successfully generalized to unlabeled data @xcite . our work is also closely related to the computational study of conversations. we build on zhang and danescu-niculescu-mizil (2020)'s unsupervised measure of forwards-range, which was originally developed to analyze strategies in counseling conversations. ## dataset we create a new open-source dataset labeled for funneling and focusing questions with the help of domain experts. we recruit former and current math teachers and educators trained in classroom observation to annotate 2,348 examples of teacherstudent exchanges. we use the same sample of exchanges as demszky et al. (2021a) -they are sampled from transcripts of 45-60 minute long 4th and 5th grade elementary math classroom observations collected by the national center for teacher effectiveness (ncte) between 2010-2013 @xcite . 2 the transcripts represent data from 317 teachers across 4 school districts in new england that serve largely low-income, historically marginalized students. transcripts are fully anonymized: student and teacher names are replaced with terms like "student", "teacher" or "mrs. h". foot_2 ## proposed methods we use a variety of supervised and unsupervised methods to identify funneling and focusing questions. ## experiments and results we evaluate the ability of our models to identify funneling and focusing questions on both the un-filtered and the filtered datasets. we train separate models on each dataset, the idea being that the model trained on the unfiltered set can help identify funneling and focusing questions in "in the wild" -i.e. in any teacher utterance, while the model trained on the filtered set can help categorize a dataset of questions as funneling or focusing. the results are shown in only few of the unsupervised measures show significant correlations with human judgments. the forwards-range has a significant negative correlation for the unfiltered set (ρ = -.130), but it changes to a significant positive correlation for the filtered set (ρ = .159). the positive correlation on the filtered set validates our assumption that focusing questions receive a greater variety of student responses. the negative correlation of the unfiltered suggests that replies to follow-up questions are less varied than other student utter-ances, which makes intuitive sense, since replies to follow-up questions may reuse words (e.g. "i think...", "yes.") and they tend to stay within the same topic as the teachers' question. the correlation pattern for length is the opposite as that of the forwards-range, showing a positive correlation with human judgments on the unfil-tered set and a negative correlation on the fil-tered set. this suggests that overall, teacher utterances containing follow-up questions tend to be longer but that focusing questions tend to be shorter than funneling ones. as for the other linguistic features, we see a significant positive correlation between the use of "why", "how do", and "what else" on the filtered set, confirming our hypotheses that indicators of open-ended questions are also indicators of focusing. in contrast, the use of "when" has a negative correlation with focusing on the filtered set, indicating that that teachers tend to use "when" when they expect a normative answer. interestingly, other question words do not show a significant correlation on the filtered set, indicating that question words in themselves are not strong indicators of funneling and focusing. question words and cognitive verbs tend to have a positive correlation with humans on the unfiltered set, which is unsurprising, as these features are all indicators of questions. overall, the trend that we see throughout the unsupervised measures is that there is not enough signal for them to reliably identify funneling and focusing questions. to measure the practical utility of our models in classroom settings, we also calculated the correlations of our model outputs with educational outcomes (see section 4.2). we conduct a similar analysis with value-added scores. since value-added scores are linked to teachers, we mean-aggregate each models' outputs at the teacher-level. then, we compute the linear correlation between each feature and the outcome. the predictions from the roberta model trained on the filtered dataset have a significant correlation with value-added scores, indicating that the measure of funneling and focusing teacher questions captures meaningful variance in students' academic outcomes. ## qualitative analysis of model outputs to better understand the performance of our models, we analyzed the predictions of our roberta model fine-tuned on the filtered set and the forwards-range. here, we choose to analyze performance on the filtered set to better understand the performance of our models in specifically distinguishing between funneling and focusing, rather than including the unfiltered set for the related but easier task of identifying if the teacher prompted the student. some selected examples that we examined are shown in the roberta model was also able to classify more complex examples that include several different question phrases. for instance, the first example in the table, includes the question phrase "how many", which correlates with funneling. but then the teacher also asks the student about their thinking, asking "what do you mean" by that, which makes the utterance an example of focusing. this suggests that the roberta model is able to account for contextual factors and weigh the importance of different question phrases. on the other hand, the forwards-range predicted this example as "funneling", which shows one of its weaknesses as a bag-of-words model that lacks context. one area of improvement across all the models we found through manual inspection is a class of focusing examples where the teacher calls on students to reflect on other students' contributions. for example, if a teacher asks a student student b, "is student a correct?", this is a closed-ended question that could be interpreted as funneling, but it is focusing since the student is reflecting on the thinking of another student. the second entry in ## conclusion and future work we propose several approaches for computationally measuring funneling and focusing, an important aspect of classroom discourse, and evaluate their strengths and weaknesses. our supervised approach using the fine-tuned roberta model has the strongest linear correlation of the models we tested with human expert ratings for funneling and focusing; it similarly had the strongest correlations of the models with educational outcomes. this shows the potential of the roberta model to be used in future feedback and professional development tools for teachers. still, our unsupervised measures show significant correlations with the expert labels for funneling and focusing, as well as with educational outcomes. this provides a foundation for future work combining different unsupervised approaches to build a robust measure of funneling and focusing. other paths for future nlp work include using probing and attention weights to better understand the predictions of the roberta model, improving the supervised approaches via an extensive hyperparameter search and by exploring models beyond roberta, and importantly, improving and testing the generalizability of this measure to other classrooms and domains. in education, there is potential for future work in exploring how this measure can best support instruction and learning outcomes for students across different educational settings. one possible avenue for this is examining discipline-specific ways of identifying focusing or funneling to provide more