Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms (2022)

Abstract: “Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similar transformer-based classification models.

## introduction there is a strong theoretical and empirical basis for encouraging students' active participation in inquiry-based and socially constructed classroom environments @xcite . numerous efforts exist to support teachers to become more purposeful and effective in their efforts to facilitate such environments @xcite . most approaches to providing teachers with detailed feedback about their discourse strategies require highly trained human observers @xcite . however, recent research has shown that the development and training of deep learning models to automate and scale certain discourse analyses from instructional episodes is feasible @xcite , effective @xcite , and reliable @xcite @xcite . accountable talk theory offers well-defined, research-based practices for teachers to engage in high-quality instruction, including the use of specific talk moves that promote students' equitable participation in a rigorous learning environment (o @xcite . by using talk moves, teachers place the "intellectual heavy lifting" and balance of talk toward students and help ensure that the discussions will be purposeful, coherent, and productive @xcite . talk moves support classroom discourse to move beyond the traditional initiate, response, evaluate linguistic sequence @xcite ; namely, by replacing the act of evaluating with practices that support a collective understanding that builds on and extends mathematical ideas @xcite .in this way, talk moves enable dialogue shifts from teacher directed recitation to true discussions in which knowledge is informally shared and constructed rather than transmitted. this paper draws inspiration from speech recognition systems for spoken dialog systems to investigate the feasibility of applying a novel input representation that utilizes tokens from previous and subsequent utterances to classify teacher talk moves @xcite . we explore three different context setups: previous-only utterances, subsequent-only utterances, and both previous and subsequent utterances (equal numbers of each) with different window sizes. in addition to the longer dialog window experiments, we re-port findings from fine-tuning transformers such as bigbird @xcite and longformer @xcite which are architected to support longer sequences. similarly, we report findings from fine-tuning mathbert, a transformer architecture that was trained to establish semantic correspondence between mathematical formulas and their corresponding context @xcite . for training and evaluation, we use the talkmoves dataset comprising 567 lesson transcripts derived from video recordings of k-12 mathematics classrooms @xcite . the main contributions of this work are summarized as follows: ## related work this section briefly describes the accountable talk theory framework, followed by a literature review on deep learning models for natural language processing (nlp) focused on adding additional contexts and learning long-term dependencies. ## method this section discusses the different approaches we took to enrich contextual cues in the talkmoves model in an effort to enhance performance. ## results in this section, we present the results from our experiments that involved providing additional context to transformers to support the process of learning long-term dependencies. the experiments were repeated with ten random seeds, and the average score is reported ( for imbalanced datasets like talkmoves, the matthew correlation coefficient (mcc) and f1 measure are good indicators of model performance. an mcc score of +1 indicates a perfect correlation while 0 indicates a random correlation and -1 indicates a negative correlation. similarly, the f1 score ranges from 0-100% where 100% indicates perfect performance. we begin with the no-context condition which achieved a performance of 71.93% f1. on prepending the immediately prior or subsequent student utterance, the model achieved a performance of 76.32% f1 @xcite . next we turn to results from various context conditions with different window sizes followed by results from longformer, bigbird, and other models. the maximum sequence length in most of these models was 512 with the exception of longformer and bigbird which had a sequence length upto 1024. the results presented in this work are comprehensive but not exhaustive since training and testing for all possible models and parameters is infeasible. the results table clearly illustrates the impor-tance of context in enhancing performance. starting with roberta-base, the performance on the previous-only condition gradually increased with an increase in window-size and saturated for larger window-sizes. similarly, we observed an improvement in performance for the subsequent-only condition. however, we did not see any significant improvement for larger window-sizes in this condition, possibly due to the negative impact in performance on "revoicing" and "restating" which rely on immediately prior student sentences. moreover, the combination of previous and subsequent utterances resulted in the best performing model. the performance gradually increased proportionally with a window size up to 7 before saturating. likewise, the performance on longformer, xlnet and bigbird were comparable with similar input representation. the most surprising result was the performance on mathbert which was signficantly lower than other models. in summary, roberta-base with equal previous-subsequent condition (@xmath0. the primary motivation of the error analysis using a confusion matrix was to improve the performance on the under-performing talk move categories and identify patterns among the misclassfied utterances to be leveraged as features for the models. when comparing the confusion matrix from prior work @xcite (see ## discussion based on the results from our experiments to improve the performance of a talk moves classifier using transformers, it is evident that longer dialog windows play an important role in differentiating talk moves. we successfully validated that the local discursive context is an important feature in classifying teacher talk moves. we generated a 4% f1 increase in performance when including a single additional utterance (either previous or subsequent) as compared to the no-context condition. also, we observed that previous utterances are more impactful than future utterances for classifying talk moves. this finding is not surprising given that several talk moves, such as the teacher "restating" and "revoicing" what a student has already said, depend entirely on previous utterances as context. we also observed that context windows with a combination of previous and future utterances outperform either condition alone. finally, we found that a window size of seven previous and subsequent utterances achieves the best performance. beyond the identified size of seven, the performance decreases. it is possible that much earlier or much later utterances provide confusing or conflicting contextual information, which hinders model performance. it is equally likely that longer dialog windows could lead to overfitting. prior efforts to address the imbalanced nature of talkmoves dataset through weighted loss resulted in reduced performance @xcite . as an alternative, we attempted to generate synthetic samples of tokenized utterances through smote (synthetic minority oversampling data) @xcite . with smote, it was challenging to retain the syntactic information of the generated examples. it was also difficult to generate the supporting contextual student and teacher utterances. preliminary efforts did not yield any improvement in performance. to further improve the performance, we have identified two future directions that appear worthwhile to consider: (1) experimenting with punctuation and other linguistic markers in the existing talkmoves dataset and (2) collecting more training data. in the talkmoves dataset, all the punctuation and other non-alphanumeric characters from the teacher and student utterances were removed. these text processing steps are typical for most text-based nlp applications to produce text that closely aligns with the output of automated speech recognition (asr) systems. however, we hypothesize that punctuation could play a significant role in differentiating one talk move from another. for example, "agreed?" with a question mark can be considered an instance of "keeping everyone together" whereas "agreed" as a statement would be an instance of "none." it remains to be determined the extent to which including punctuation markers might impact the performance of the models. similarly, we can try incorporating speaker turns to indicate a student or teacher turn in previous and subsequent utterances as additional features to the model. another option that warrants consideration is supplementing data for the purpose of model pretraining. talkmoves dataset (github.com/sumnerlab/talkmoves) is a relatively small dataset for pretraining transformers when compared to roberta which was pretrained on millions of data points. at the same time, we recognize the challenge in the collecting and annotating thousands of classroom transcripts. moreover, there are important privacy concerns and other ethical considerations, given that these data involve minors, use proper names (which can be critical information for talk moves classification), and can be challenging to access in large quantities. we could potentially explore active learning to achieve greater accuracy with limited samples @xcite . active learning is often sought as an option in machine learning applications where unlabeled instances are abundantly available @xcite . ## conclusion documenting consequential elements of classroom instruction and providing teachers with feedback on their practices are critical endeavors in the education field. taking into consideration the strong need to provide reliable feedback to teachers on productive classroom discourse, we need robust models to automatically classify teacher talk moves with high reliability. in this paper, we report on a number of experiments that involved providing longer dialog windows to the transformers in an effort to improve model performance. based on these experiments, we generated a state-of-the-art 2.6% f1 improvement in performance (78.92% f1) over the previous models, primarily by adding a set number of previous and subsequent utterances to the input representation. clearly, there are both challenges and opportunities for the development of innovative uses of ai techniques, particularly as they can be incorporated into tools that support teacher and student learning. the findings from this research open new avenues for exploration that can benefit both the education and nlp communi-