Virtual Pre-Service Teacher Assessment and Feedback via Conversational Agents (2021)

Abstract: Conversational agents and assistants have been used for decades to facilitate learning. There are many examples of conversational agents used for educational and training purposes in K-12, higher education, healthcare, the military, and private industry settings. The most common forms of conversational agents in education are teaching agents that directly teach and support learning, peer agents that serve as knowledgeable learning companions to guide learners in the learning process, and teachable agents that function as a novice or less-knowledgeable student trained and taught by a learner who learns by teaching. The Instructional Quality Assessment (IQA) provides a robust framework to evaluate reading comprehension and mathematics instruction. We developed a system for pre-service teachers, individuals in a teacher preparation program, to evaluate teaching instruction quality based on a modified interpretation of IQA metrics. Our demonstration and approach take advantage of recent advances in Natural Language Processing (NLP) and deep learning for each dialogue system component. We built an open-source conversational agent system to engage pre-service teachers in a specific mathematical scenario focused on scale factor with the aim to provide feedback on pre-service teachers’ questioning strategies. We believe our system is not only practical for teacher education programs but can also enable other researchers to build new educational scenarios with minimal effort.

## introduction in the era of remote teaching due to governmental regulations and stay-at-home orders for covid-19, remote teaching methodologies have come to the forefront of education and training. virtual conversational agents have been used for a variety of * * first two authors have contributed equally training scenarios to train teachers and nurses in many different contexts @xcite . virtual conversational agents refers to an online interactive system where a user is able to dialogue and receive responses in turn. all future references to conversational agents will refer to this definition of virtual conversational agents. conversational agents, often characterized as dialogue systems, are common in customer support representative applications @xcite . in education, they have been used extensively for interpreting the content of the conversation, integrating and assimilating information, and providing feedback @xcite . however, as pointed out by @xcite very few conversational agents used in education use recent advances in machine learning and deep learning, instead relying on simple decision trees. this reinforces the need for more research and development on artificial intelligence-based methods to support content-specific conversations. a conversational agent deployed through a web interface, as opposed to software implementations, for teacher learning has multiple benefits. two of the prominent benefits are that this approach can easily be scaled to reach more in-service and pre-service teachers as well while also providing a cost-effective way to build systems that can be scaled to new teaching scenarios as multiple dialogue system components can be reused. the dialogue manager and semantic sentence-level components can be used for different mathematical scenarios as long as the assessment component remains unchanged. in this case the term assessment is referring to pre-service teacher evaluation as opposed to a students understanding of a mathematical topic. in this work, we combine the advances in nlp and deep learning research with a modified version of the instructional quality assessment (iqa) @xcite framework to build a scenario to be used in teacher education settings. our goal in this project is to assess and give teachers immediate feedback on the quality of their instructional moves using a specific scenario on a given mathematical topic. research demonstrates that the instructional quality assessment provides a robust framework for evaluating teachers' instructional practice in mathematics classrooms. our demonstration uses a modified version of the iqa that focuses on the strategy of teachers' questions during one-on-one or class discussions through a web-based platform that allows for pre-service teachers to receive realtime feedback on the quality of their questioning. although our demonstration centers on using an adapted version of iqa as the assessment component, our modular architecture can be used to incorporate alternative evaluation schemes as well. a critical challenge of developing dialogue systems in a new domain is the requirement of large data sets for the different components of the dialogue system (intent classification, slot filling for dialogue state tracking, and the dialogue policy). in education, where data collection can itself prove a significant challenge, especially given a need for increased domain expert annotations, our system is developed with only a small amount of annotated data: around two thousand sentences that are labeled with one of four adapted iqa classes. note that this system is not necessarily better than existing approaches that use annotated training data for each component of the dialogue system, rather, this system is a compromise because dialogue systems are challenging to build in low-or no-data amount scenarios. what our system demonstrates is a solution to developing a highly-manipulable scenario given minimal domain-expert annotated data that can be used to support virtual feedback for pre-service teachers. our contributions are as follows: ## related work virtual human-based simulations can provide meaningful, deliberate practice for learning a wide range of teaching skills during teacher preparation as well as extended practice for advanced skills for in-service teachers. dialogue systems for conversational agents can be built in two different approaches: end-to-end approaches that combine all of the dialogue system stages require large-scaled labeled training data and component-based systems which require less data, but each component needs to be trained separately. component-based systems have the advantage of allowing certain components to be reused for similar contextual scenarios. our work utilizes the latter approach leveraging unstructured data collected from web and textbooks as knowledge bases. our dialogue policy is a combination of handcrafted rules by education domain experts, and dialogue states are tracked through a reading comprehension based approach highlighted by @xcite . the most common approach for dialogue state tracking is the "slot-value" pairs approach. in this scenario, different stages of the dialogue are often framed as a multi-class classification task @xcite . while this approach is well-studied, the decision to use the reading comprehension based approach in this system is based on the ability to incorporate pre-trained models for dialogue state tracking. the assessment component for the pre-service teachers' questioning strategy is an adapted version of iqa. the iqa has been developed by the learning research and developmental center at the university of pittsburgh since 2002 @xcite . the iqa offers a holistic assessment of mathematical instruction, including the academic rigor of the specific tasks and student and teacher discussion surrounding the task @xcite . the iqa has been further validated by its developers in subsequent research @xcite . more recent developments suggest that the iqa can be used not only as an assessment tool but also as a feedback tool to help teachers actively improve their instruction @xcite . the questions that teachers ask are essential for promoting students' meaningful mathematical discourse. the academic rigor component @xcite of the iqa builds on earlier classifications of teacher questions (e.g. @xcite , distinguishes between "probing and exploring" questions that ask students to clarify their ideas or the connections between them, and "procedural and factual" questions that elicit facts or yes/no responses). the iqa is intended to be used in contexts where cognitively demanding mathematical tasks are implemented and is well suited for fine-grained teacher professional development such as that which focuses on teacher questioning @xcite . another advantage of utilizing iqa is the limited categories as a high number of categories results in very complex dialogue policies @xcite . given the limited amount of time domain experts may have for annotating data, several methods to improve label efficiency were explored. weak supervision techniques, as highlighted by @xcite provides the two-fold benefit in that it requires less human labeling than would otherwise be required for training. an additional benefit of weak supervision is that noisy data and the accuracy of each annotator can be taken into account for classification. @xcite has shown that weak supervision systems are better than generic majority vote approaches. noisy label data for model classification has also been studied by deeplearning-based approaches @xcite and proven effective. ## methods the overall architecture of our conversational agent is depicted in figure 2 . as discussed, a central component of our demonstration is evaluating and providing feedback of pre-service teacher instruction. for this initial version of the scenario, domain experts provided a specified rubric for pre-service teachers to meet that clarifies the types of iqa categories desired within a session. one sample rubric evaluates if the teacher is asking at least one "probing and exploring" question, one "expository and cuing" question, and one "procedural and factual" question. this rubric can be changed easily in the demonstration through a separate json file. in our current evaluation, we do not evaluate the order in which the questions are asked, but we plan to include more sophisticated evaluation protocols in future work. the current implementation incorporates text interactions between a conversational agent, representing a student, and a pre-service teacher. the pre-service teacher can interact with the conversational agent by providing new knowledge and testing understanding as well as testing knowledge of the topic "scale factor". the pre-service teacher types a statement, question, multiple statements or multiple questions in the text box, and the conversational agent responds by taking into account the conversation context as well as the pre-service teacher's utterance. each component of the dialogue system is described in detail in the subsequent subsections. ## results this section will discuss the step by step interaction with the developed interface that is developed. only one scenario currently exists and is defined as a 4th-5th grade average understanding of the mathematics topic scale factor. ## conclusion and future work our goal in this paper is to demonstrate the implementation of a conversational agent with very little training data that relied on foundational and well-studied metrics like iqa. by leveraging state of the art modules for natural language processing and deep learning we could build a functional prototype that is now going to be used as a pilot for training for a specific mathematical scenario of "scale factor". by integrating pre-trained models such as squad, bert, and the universal sentence encoder as well as using weak supervision approaches in data treatment we have leveraged minimal amounts of domain-expert-labeled data and knowledge base data in order to create a usable interface. approaches like this help evaluate pre-service teachers in a scalable fashion and can also be deployed across the web for large scale participation. there are several areas we are pursuing to improve this interface in order to provide a more robust interface as well as a more useful assessment tool. some of the current features that we are working on are as follows: finally, we plan to be deploy this tool with a group of pre-service teachers under the direction of expert teachers in order to test the qualitative aspects and realism of this system. figure 6: conversational agent session example: acknowledgement