NAIST eacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues (2023)

Abstract: This paper presents our approach to the BEA 2023 shared task of generating teacher responses in educational dialogues, using the Teacher-Student Chatroom Corpus. Our system prompts GPT-3.5-turbo to generate initial suggestions, which are then subjected to reranking. We explore multiple strategies for candidate generation, including prompting for multiple candidates and employing iterative few-shot prompts with negative examples. We aggregate all candidate responses and rerank them based on DialogRPT scores. To handle consecutive turns in the dialogue data, we divide the task of generating teacher utterances into two components: teacher replies to the student and teacher continuations of previously sent messages. Through our proposed methodology, our system achieved the top score on both automated metrics and human evaluation, surpassing the reference human teachers on the latter.

## introduction the shared task for bea2023 was to generate teacher utterances in an educational dialogue, specifically one between an english language learner and their language teacher @xcite . the data was collected from one-on-one english lessons between real teachers and students conducted over a chat application. the data for the task consists of fragments of these dialogues, with the goal of predicting the next teacher utterance. inspired by a commonly used practice in machine translation @xcite @xcite , our system generates multiple candidates and reranks them. given the high level of fluency required for this task, we began with a pretrained language model (gpt-3.5turbo) rather than training one from scratch. an overview of the system is outlined in figure 1 . first the prompts for the prompt ensemble are chosen based on the conversation history. the prompt ensemble is sent to gpt-3.5-turbo to generate a set of candidate responses. during a postprocessing step some responses are flagged as inappropriate for referring to pronunciation, responding as a student, or containing profanity. these inappropriate responses are filtered out before reranking. the response chosen by the reranker is returned. our prompt ensemble consists of a mixture both zero and few shot prompts. for the few shot prompts, we experimented with several different ways of selecting the examples to provide, but the most effective was an iterative approach that, to our knowledge, is novel. this approach works in two steps: the first step is to generate the candidate teacher utterances for all conversations using the zero-shot prompt and score them. in the second step, the highest and lowest scoring responses are integrated into the prompt as positive and negative example responses. to generate multiple candidates, we instructed the model to return multiple possible responses with a single prompt similar to @xcite . we found this to be a very effective technique because it resulted in a diverse set of candidates to select from in the reranking stage. we tested our methods on a held-out section of the training data which contained reference responses using bertscore @xcite and dialogrpt @xcite . we found the iterative few-shot approach increased the overall score of the data on both automated metrics. however, manual inspection of the output showed that this approach works better for some cases than others. teacher response or teacher continuation? as the data was collected from real-time chats between real people, the turns did not alternate consistently as they do in most dialogue systems. the task of generating a teacher continuation -that is, a teacher follow-up message when that same teacher was the last to have taken a turn in the conversational history -was particularly challenging. this phenomenon occurs in 38% of the training data, and 40% of the development data. as systems are trained to alternate turn taking, this led to many of the generated responses erroneously sounding like they were either uttered by the student, or showed a lack of awareness of the immediately preceding utterance. figure 2 shows one such conversation and an example of a problematic generated response. the teacher first asks the student to look at the text and think about if the word "startling" means obvious or surprising. the ai generated follow-up gives the answer away immediately ("startling" means surprising), which overrides the original goal: to encourage the student to find the answer themselves. to address this, we split the data into two tasks based on who spoke last: teacher reply for when the context ended with the student, and teacher continuation for when the context ended with the teacher. we evaluated each task separately during development to design a set of prompts for each. ## data although we explored augmenting with data from other sources, in the final system we only used the data made available for this shared task, the teacher-student chatroom corpus @xcite . the training data consists of 2747 partial conversations between a student learning english and their teacher. we split out a random sample of 275 conversations (10%) to use as validation data. our experiments were carried out on this split because the development data supplied by the shared task did not contain reference answers. student ...provocative, and supplied +startling+ messages about psycology... student is it enough this piece of text to work on it? the word i don't konw is startling teacher ok thanks....'startling' does it mean: obvoius or surprising do you think? is there a context in the text to help you? teacher great job using the keywords from the text! "startling" means surprising. in this context, it seems like the text contains provocative and surprising messages about psychology. can you think of any other words that might be related to the topic of psychology? besides a single run of our system on this data to submit to codalab, we did not use the 305 conversations in the supplied development dataset. the test data contained 273 conversations. ## system our system generates a number of responses by prompting gpt-3.5-turbo using the default parameters, and chooses among them using a dialogrpt-based reranker. naisteacher uses several different prompts, ranging from general to those targeting specific scenarios. ## results despite never using the reference sentences for training or fine-tuning, our system received the highest bertscore and second highest dialogrpt score on the evaluation data, giving us the highest average rank in the automated metrics. the human evaluation of the top three teams was carried out on the prolific crowdsourcing platform, where our system was compared against the other two systems as well as against the reference teacher utterances. the raters chose the best response on three criteria: (1) which was more likely said by a teacher, (2) which shows a better understanding of the student, and (3) which helps the student more. our system scored the highest out of the three manually evaluated systems on all three criteria, and especially highly on the helpfulness criteria. ## conclusion the reranker we built was very simple. it selected the highest-scoring response according to the automated metric, dialogrpt. however, preliminary manual evaluation did not always align with dialo-grpt. the metric often prefers complete answers that do not encourage student engagement over responses that aim to help the student answer the question for themselves. with additional time, we would like to develop a model capable of classifying the extent to which a generated response reflects a teacher's style. this model could take into account whether the response effectively balances helpfulness with scaffolding independent thought, as well as the degree to which it demonstrates an understanding of the student's needs. such a model would lead to improved performance of the reranker. the ai teacher response generator we created still needs improvement before it can become a fully functional teacher chatbot. the responses it generates can be excessively detailed at times due to the automated metrics used, which prioritize comprehensive responses. when integrated into an assistant, it may seem as if the responses are repetitive or that there is no well-designed lesson plan in place. as we were building the system, we kept in mind how it was to be evaluated. that is, by machine first checking for similarity to a reference answer as well as usefulness and relevance to the conversation, then by humans evaluating how teacher like, understanding, and helpful the response was. we used a combination of @xmath1, and @xmath2. while the automated metrics taken together align with human evaluators' judgments, dialogrpt alone does not always correspond with human judgment. the dialogrpt score models were trained on reddit, which follows a different format than live chat. reddit is an asynchronous format, meaning that it tends to have longer, more complete responses. on the other hand, synchronous chatbased lessons feature multiple consecutive turns, as it is more common in instant messaging to break up longer thoughts into smaller turns. dialogrpt was not trained to judge the continuation of a response, which made it less reliable as a reranker for teacher continuations, in particular. when chatting, it's not necessary to always respond with a detailed message. the reference teacher responses offer a mix of quick replies, corrections, elaborations, practice activities, and clarifications, among others. in the future, we would like to incorporate more of the conversational moves that real teachers use in these types of exchanges. in conclusion, our approach to generating teacher utterances in an educational dialogue for the bea2023 shared task used a pretrained language model and an ensemble of prompts to generate multiple candidates, which we then reranked using automated metrics. we experimented with different techniques for generating few-shot prompts and found that an iterative approach was the most effective. our system achieved the highest averaged ranked scores in both the automated and human evaluation rounds. overall, our approach shows promise for generating effective and helpful teacher utterances in educational dialogues.