The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues (2023)

Abstract: This paper describes the results of the first shared task on generation of teacher responses in educational dialogues. The goal of the task was to benchmark the ability of generative language models to act as AI teachers, replying to a student in a teacher-student dialogue. Eight teams participated in the competition hosted on CodaLab and experimented with a wide variety of state-of-the-art models, including Alpaca, Bloom, DialoGPT, DistilGPT-2, Flan-T5, GPT- 2, GPT-3, GPT-4, LLaMA, OPT-2.7B, and T5- base. Their submissions were automatically scored using BERTScore and DialogRPT metrics, and the top three among them were further manually evaluated in terms of pedagogical ability based on Tack and Piech (2022). The NAISTeacher system, which ranked first in both automated and human evaluation, generated responses with GPT-3.5 Turbo using an ensemble of prompts and DialogRPT-based ranking of responses for given dialogue contexts. Despite promising achievements of the participating teams, the results also highlight the need for evaluation metrics better suited to educational contexts.

## introduction conversational ai offers promising opportunities for education. chatbots can fulfill various roles -from intelligent tutors to service-oriented assistants -and pursue different objectives such as improving student skills and increasing instructional efficiency @xcite . one of the most important roles for an educational chatbot is that of an ai teacher which helps a student improve their skills and provides more opportunities to practice. recent studies suggest that chatbots have a significant effect on skill improvement, for example, in language learning @xcite . moreover, the advances in large language models (llms) open up new opportunities as such models have a potential to revolutionize education and significantly transform learning and teaching experience. despite these promising opportunities, the use of powerful generative models as a foundation for downstream tasks presents several crucial challenges, in particular, when such tasks may have real social impact. specifically, in the educational domain, it is important to determine how solid that foundation is. @xcite stresses that if we want to put such models into practice as ai teachers, it is of crucial importance to determine whether they can (a) speak to students like a teacher, (b) understand students, and (c) help students improve their understanding. following these desiderata, @xcite formulated the ai teacher test challenge: how can we test whether state-of-the-art generative models are good ai teachers, capable of replying to a student in an educational dialogue? building on the ai teacher test challenge, we have organized the first shared task on generation of teacher language in educational dialogues. the goal of this task is to explore the potential of nlp and ai methods in generating teacher responses in the context of real-world teacher-student interactions. interaction samples werer extracted from the teacher student chatroom corpus @xcite , with each training sample consisting of a dialogue context (i.e., several rounds of teacherstudent utterances) and the teacher's response. for each test sample, participants were asked to submit their best generated teacher response. as the purpose of this task was to benchmark the ability of generative models to act as ai teachers, responding to a student in a teacher-student dialogue, submissions were first ranked according to popular bertscore and dialogrpt metrics, and the top three submissions were then selected for further human evaluation. during this manual evaluation, the raters compared a pair of "teacher" responses along three dimensions: speaking like a teacher, understanding a student, and helping a student @xcite . speaker utterance teacher: yes, good! and to charge it up, you need to __ it ___         dialogue context student: . . . teacher: connect to the source of electricity student: i understand teacher: plug it __? student: in teacher: yes, good. and when the battery is full, you need to ____ (disconnect it) = reference response 2 materials and methods the shared task used data from the teacher-student chatroom corpus (tscc) @xcite . this corpus comprises data from several chatrooms in which an english as a second language (esl) teacher interacts with a student in order to work on a language learning exercise and assess the student's english language proficiency. ## results the results achieved by the participating teams during the automated evaluation phase are shown in as can be observed from the adaio system @xcite attained the second-best average rank on both the automated evaluation phase ( it is interesting to note that the teacher's response was ranked lower than the top three systems built on gpt-3 and gpt-4 (figure 4 ), which contradicts the results of @xcite . this striking observation might be explained by some differences in the human evaluation procedure: while any native english speaker could participate in @xcite , only raters working in education and training could participate in the shared task. some of these raters gave specific feedback stating that they found the non-standard language used by the teacher in the chatroom (including spelling mistakes, typos, and such) unprofessional. for more in-depth analyses, the reader is referred to the system papers cited in this paper. in these papers, the participating teams ran additional analyses and made critical observations. for example, @xcite observed that fine-tuned models attained better results on bertscore, prompting attained better results on dialogrpt, and methods that combined both techniques showed competitive results across all metrics. at the same time, they found that a baseline generating "hello" in response to every prompt achieved the best result for bertscore precision and dialogrpt updown. @xcite found that gpt-2 -a smaller model with 124 million parameters -achieved competitive performance compared to the t5-base model. moreover, they found that, even though they maximized bertscore f1 as a reward function, their model scored highly in terms of the other evaluation metrics. @xcite noted that dialogrpt often preferred complete answers that were not very teacher-like over responses that helped the student find the answer by themselves. ## discussion although the inaugural shared task on generating ai teacher responses in educational dialogues can be considered a success, the results demonstrate that the evaluation of natural language generation models remains challenging. ultimately, we would like to have at our disposal precise, valid, andideally -automated methods that reward machines and/or humans for their pedagogical abilities. however, we are probably still a long way from achieving this ultimate goal. the automated metrics that currently exist are not capable of rewarding models for their ability to showcase pedagogical skills. in particular, to the best of our knowledge, there does not exist any comprehensive metric capable of evaluating whether responses are likely to be produced by a teacher, as well as whether they demonstrate understanding of what the student is saying and are helping the student. moreover, popular automated metrics such as bertscore and dialo-grpt used in this task show a considerable sensitivity to construct-irrelevant variations, as is demonstrated by the use of a "hello" baseline @xcite and an inclusion of the "teacher:" prefix @xcite . future editions of this task should, therefore, aim to either develop or resort to more accurate and domain-specific automated metrics as per the observations and suggestions from several competing teams @xcite @xcite . due to the lack of adequate metrics, we need to resort to manual evaluation methods in order to achieve more precise assessments. however, a typical drawback to manual evaluation is that it is very costly and time-consuming to have a sufficient number of raters evaluating any possible response that can be generated in the large space of possible teacher replies. due to practical and budgetary limitations, it is challenging to organize a shared task during which any possible number of submissions can in principle be evaluated with adequately remunerated human evaluations. what is more, data is very important in the context of real-world applications and shared tasks. although the corpus used in this shared task is a valuable resource in our domain, some particularities of this corpus and the data sampling method also had an undeniable impact on the results. therefore, in future editions of this shared task we should rethink some of the current potential limitations, such as the fact that the dialogues had to be limited to 100 tokens, resulting in partial conversations; the fact that some dialogues, if extracted from the data randomly might have led to data leakage; and the fact that the dialogues did not always follow strictly role-alternating format, with some teacher turns being preceded by previous teacher utterances, rather than a student utterances. in summary, the field of education has already been significantly changed by llms, whose capabilities keep improving constantly. we hope that this shared task will serve to help the scientific community better understand the current capabilities of llms in educational contexts. having learned from this shared task and going forward, we hope to make its future iterations even more informative. ## conclusion the primary goal of this shared task was to explore the potential of the current state-of-the-art nlp and ai methods in generating teacher responses in the context of real-world teacher-student interactions. a number of diverse and strong teams participated in the task and submitted outputs of their systems to the competition, and even more people expressed their interest. the teams used a variety of the state-of-the-art large language models and explored diverse prompting and fine-tuning approaches. importantly, these results not only shed light on the current state-of-the-art on this task but also highlighted some critical limitations that should be addressed in the future.