Discussion Tracker: Supporting Teacher Learning about Students’ Collaborative Argumentation in High School Classrooms (2020)

Abstract: Teaching collaborative argumentation is an advanced skill that many K-12 teachers struggle to develop. To address this, we have developed Discussion Tracker, a classroom discussion analytics system based on novel algorithms for classifying argument moves, specificity, and collaboration. Results from a classroom deployment indicate that teachers found the analytics useful, and that the underlying classifiers perform with moderate to substantial agreement with humans.

## introduction collaborative argumentation in student dialogue is essential to individual learning as well as group problem-solving @xcite . strong collaborative argumentation is characterized by specific claims, supporting evidence, and reasoning about that evidence as well as by building upon, questioning, and debating ideas posed by others. however, teaching collaborative argumentation is an advanced skill that many high school teachers struggle to develop @xcite , partially due to the practical challenge of keeping track of important features of students' talk while managing class and reflecting on students' talk when no record of it exists. to address this challenge, we have developed discussion tracker (dt), a system that leverages natural language processing (nlp) to provide teachers with automatically generated data about three important dimensions of students' collaborative argumentation: argument moves, specificity and collaboration. discussion tracker includes visualizations, interactive coded transcripts, collaboration maps, analytics across discussions, and instructional planning. in contrast to teacher dashboards which largely focus on discussion analytics such as amount of student/teacher talk, teacher wait time, and teacher question type @xcite @xcite , dt focuses on students' collaborative argumentation. in contrast to related nlp algorithms which largely focus on coding student essays @xcite @xcite , asynchronous online discussions @xcite , and news articles (li and nenkova, 2015), dt's nlp algorithms address the challenges of coding transcripts of synchronous, face-to-face classroom discussions. ## description of discussion tracker (dt) to use dt, a teacher first uploads a classroom discussion transcript. next, nlp classifiers code the transcript using a previously developed scheme for representing three important dimensions of collaborative argumentation @xcite : argument moves (claim, evidence, explanation), specificity (low, medium, high), and collaboration (new, agree, extension, challenge/probe). student turns are the unit of analysis for collaboration. argumentative discourse units (adus) -either entire turns, or segments within turns -are the argumentation and specificity units of analysis. each nlp classifier in dt was developed by training on a previously collected and freely available corpus 1 of collaborative argumentation @xcite using transformer-based neural networks. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . all models use the bert-base-uncased bert variant from the huggingface (wolf et al., 2019) library, which results in the smallest available dimensionality to keep computational complexity to a minimum. the three models were built using the keras library (chollet and others, 2015). the adam optimizer was used, as well as early stopping to automatically determine the number of epochs for training by monitoring validation loss (the validation set was chosen randomly and consisted of 10% of the initial training set for each fold). after classification, all discussion analytics are automatically generated from the nlp codes. the dt overview screen (figure 1 ) includes pie charts indicating the distribution of the codes for students' argument moves, specificity, and collaboration. other screens include interactive coded transcripts (figure 2 ), collaboration maps (figure 3 ), identification of strengths and weaknesses to support teacher goal-setting (figure 4 ), and a history page (not shown) that compares the code distributions across discussions. we initially implemented a desktop version of dt using python and tkinter. the screenshots in the figures and the usability evaluation below are based on this version. to make dt more portable across hardware and to allow teachers to easily use dt on multiple machines (e.g., school, home), we now have a web version of dt foot_0 . this version is implemented in python and uses the remi package foot_1 to convert python into html and launch a webserver to accept requests for the site and handle user input. with this setup it is easy to integrate the classifiers, implemented as a rest api on the same server hosting . ## evaluation from january to march 2020, we collected data (corpus c2) to evaluate both teacher perceptions of dt as well as nlp classifier performance. in particular, the desktop version of dt was used by 18 high school english language arts teachers from 4 schools, where: 1) each teacher led a discussion about a literary text that was audio-recorded and observed by a researcher, 2) each teacher completed an online survey within a day, 3) experienced annotators foot_2 hand-coded transcripts of the discussion for the three dimensions of collaborative argumentation discussed above and uploaded them into the dt system, 4) within two weeks, researchers conducted a 45-minute cognitive interview @xcite with each teacher while they were using dt to look at their students' discussion foot_3 , and 5) the same day, teachers completed a second survey that mirrored the first with additional items for ratings of dt. dt usability. we measured teachers' perceptions of the overall usefulness of dt and of specific features/visualizations through likert-scale items on the survey from step 5 above. survey items were based on @xcite teacher survey of perceived usability of technology. to remove noise that might distract from this usability evaluation, we evaluated dt under the best possible nlp conditions by using the manual codings of collaborative argumentation from step 3 above to generate all analytics. the nlp codings are separately evaluated in the classifier discussion below. ## summary and future directions in this work we described the development of a classroom analytics system and reported usability results from real world classroom deployment. we conducted a survey that showed teachers found the system easy to use and the analytics (based on human-annotated labels) helpful in analyzing collaborative argumentation. evaluation of the automated nlp classifiers showed that they are in moderate to substantial agreement with the labels provided by human annotators. the main goal of future work is to continue to enhance our neural classification methods, and to develop an end-to-end, completely automated system. to this end, we will consider several aspects: perform new data collections and improve classifier performance; incorporate automatic speech recognition to perform automated transcription; develop algorithms to automatically segment turns into adus. in addition, we will further develop the interface by addressing teacher feedback and improving the system's ease of use. finally, teachers will evaluate our newer versions of dt, including versions where the analytics are based on classifier outputs rather than human-annotated labels.