The ADAIO System at the BEA -2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues (2023)

Abstract: This paper presents the ADAIO team’s system entry in the Building Educational Applications (BEA) 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues. The task aims to assess the performance of state-of-the-art generative models as AI teachers in producing suitable responses within a student-teacher dialogue. Our system comprises evaluating various baseline models using OpenAI GPT-3 and designing diverse prompts to prompt the OpenAI models for teacher response generation. After the challenge, our system achieved second place by employing a few-shot prompt-based approach with the OpenAI text-davinci-003 model. The results highlight the few-shot learning capabilities of large-language models, particularly OpenAI’s GPT-3, in the role of AI teachers.

## introduction the current success of large language models (llms) in generating natural language responses that are almost indistinguishable from that of a human indicates that ai systems are steps closer to passing the turing test. apart from being used as conversational agents, llms can be employed in various educational settings as described in @xcite including as an ai teacher to help students practice and improve. @xcite launches a shared task at the 18th workshop on innovative use of nlp for building educational applications (bea 2023), called generating ai teacher responses in educational dialogues. inspired by @xcite , this task requires teams to develop intelligent tutoring systems (its) that generate teacher responses in real-world teacher-student interactions. this task serves as a benchmark to gauge the capability of generative models in functioning as ai teachers. dialogue-based its face various requirements and challenges in meeting the needs of effective educational support. this entails generating factually accurate content and ensuring educational efficacy by speaking to students in a teacher-like manner, understanding their needs, and helping them improve their understanding @xcite . however, several challenges must be addressed. one significant challenge lies in acquiring appropriate data for training its, particularly real teacher-student interactions that cover various subjects. another challenge involves developing models that can effectively capture the student's learning style and accommodate long-range dependencies within conversational sequences. furthermore, evaluating the quality of teacher responses is essential. the responses should not only sound natural but also demonstrate an understanding of the student's queries and provide valuable guidance to help the student improve. ## related work research on intelligent tutoring systems has spanned many decades, with various proposed systems that include both text-based @xcite , spoken dialogue tutoring systems @xcite and multi-modal systems that have been developed to improve student learning. earlier dialogue-based its were designed using rule-based cognitive modelling methods @xcite in generating teacher responses. in recent years natural language generation (nlg) tasks generally benefited from models using sequence-to-sequence architectures @xcite . current state-of-the-art models such as openai gpt-3 @xcite have shown tremendous results on a range of downstream nlg tasks such as response generation. one of the major underlying components of the language model is the transformer architecture @xcite which increases its capacity for context awareness and long-range dependencies. currently, the application of llms within the educational domain @xcite indicates they could improve student learning outcomes. however, their efficacy in conversational tutoring has not been fully evaluated @xcite . on bench-marking the efficacy of llms in generating responses to accomplishing teaching goals, @xcite investigate the suitability of these ai-teacher responses by comparing text generated by state-of-the-art models, blender @xcite and gpt-3, on real-world tutoring dialogue data. the paper comparatively analyses the responses based on a stack of evaluation methods. furthermore, the paper suggests the following pedagogical dimensions to evaluate the ai-teacher generated responses, on its ability to speak like a teacher, understand a student and help a student. these dimensions form the core of the ai-teacher challenge. the dataset includes a train set of 2,747 dialogues with an average of 3.9 turns per dialogue (±2.2, @xmath0. the dev set consists of 305 dialogues with an average of 4.0 turns (±2.2, @xmath1, while the test set comprises 273 dialogues with an average of 2.6 turns (±1.5, @xmath2. the response lengths in the train set range from 1 to 66 words, with an average of 9.1 words (±8.2) whereas the dev and test sets are without the response data. ## discussion the evaluation results from both machine and human assessments of the generated responses on the test set provide evidence of the effectiveness of llms, particularly gpt-3, in tutoring dialogue applications. while the dialogues in the tscc corpus primarily concentrate on everyday speech and language usage, which proves advantageous for short conversational exchanges such as corrections, explanations, or clarifications, it is crucial to examine the gpt-3 model's reliability in tutoring scenarios that involve longer sequences within a wider discourse context @xcite . furthermore, we perceived a limitation in relying solely on automatic evaluation metrics (as detailed in section 5.1) prompt engineering to adapt language and tone in tutoring systems our experiments reveal an intriguing finding where manipulating the prompt influences the tone and language of the generated response, presenting an opportunity for tutoring systems to potentially adapt to the students' learning styles and/or teaching goals. further research should delve into teaching instruction methods, potentially exploring the pedagogy of constructivist learning @xcite or engaging students in ill-structured exercises for productive failure @xcite using llms of this nature. gpt-3's robust handling of errors and noncanonical form of language during the data preparation phase, a manual inspection of the data revealed the presence of grammatical and spelling errors in some utterances. additionally, since the dataset originated from chatroom text-based conversations, there were instances where mathematical symbols were used instead of natural language, such as this example utterance output teacher: but e.g. pleased with their @xmath3. it is worth noting that we did not employ any nlp processing toolkit to correct these errors or non-canonical forms in the dialogue utterances. however, despite this, the gpt-3 model could still generate appropriate responses effectively. llms' potential in multilingual settings in the context of l2 acquisition, the dialogue nature in @xcite provides valuable opportunities for tutors to adapt to students' native languages. code-switching strategies as such have been found to enhance teaching, including the explanation of concepts @xcite , and leveraging ai tutoring systems can facilitate this process. llms possess multilingual capabilities that enable them to address language barriers, accommodate low-resource languages, and exhibit promising performance even on unseen languages @xcite . to enhance accessibility, the development and adoption of open-source multilingual models, such as bloom @xcite , should be encouraged, thereby facilitating the utilization of llms in educational applications across diverse linguistic contexts. ## conclusion in this paper, we have presented our system entry to the bea 2023 shared tasks on ai-teacher response generation. our approach investigates the capability of the state-of-the-art language generative model, openai gpt-3, in addressing the requirements of the ai teacher challenge outlined by tack and piech 2022. through extensive experimentation utilizing zero-shot, few-shot, and finetuning techniques, we investigated the adaptability of the system's responses by leveraging meticulously designed prompts and carefully selected dialogue examples that emphasize desirable teacher qualities. our submitted system, featuring a fewshot prompt-based method, achieved 2nd place in the bea shared task 2023 challenge. table continued from previous page field content history student: informally you hear people say e.g. so i go to the yoga class and it's so hard etc do that people mean that they go everyday or everyweek and suffer? teacher: by the way, the structure: 'i changed my idea' is a bit weird...normale.g. it (the experience) changed my idea student: i changed my mind would sound better?? ref. responses teacher: ah well that would be normal in that meaning but no -i mean to describe a specific past situation prompt a teacher: yes, "i changed my mind" would be a better way of expressing it. as for the yoga class, it could mean that they attend the class on a regular basis and find it challenging. prompt b teacher: yes, "i changed my mind" would sound better. to answer your question, it depends on the context. it could mean that they go to the yoga class regularly and find it challenging, or it could mean that they went once and found it difficult. prompt c teacher: yes, "i changed my mind" would sound better. as for the yoga class, it depends on what the person is saying -it could mean they go every day, or it could mean they go occasionally and find it difficult.' prompt d yes, "i changed my mind" would sound better. itś important to use the right grammar when speaking english. but, to answer your question, it depends on the context of the conversation. people might mean that they go to the yoga class every day or every week and find it difficult, or they might just mean that it was difficult that particular time. prompt e speakerb: yes, 'i changed my mind' would sound better :)