RETUYT - I n C o at BEA 2023 Shared Task: Tuning Open-Source LLM s for Generating Teacher Responses (2023)

Abstract: This paper presents the results of our participation in the BEA 2023 shared task, which focuses on generating AI teacher responses in educational dialogues. We conducted experiments using several Open-Source Large Language Models (LLMs) and explored fine-tuning techniques along with prompting strategies, including Few-Shot and Chain-of-Thought approaches. Our best model was ranked 4.5 in the competition with a BertScore F1 of 0.71 and a DialogRPT final (avg) of 0.35. Nevertheless, our internal results did not exactly correlate with those obtained in the competition, which showed the difficulty in evaluating this task. Other challenges we faced were data leakage on the train set and the irregular format of the conversations.

## introduction nowadays, with the important development of large language models (llm) and their great generative power, the interest in the development of chatbots that simulate interactions between humans has increased. in particular, in the educational domain, the use of chatbots seems to have interesting benefits, such as their potential for adaptive learning, tailored to each student, or their permanent availability @xcite . the contributions of these tools to learning are not yet clear @xcite . in their review of the area, these authors conclude that the development of chatbots is usually based on technological criteria, but the focus has not yet been placed on their pedagogical contributions in terms of learning improvements. however, there is some evidence that for language learning in particular, these tools bring certain benefits @xcite , mainly for students at initial levels. it should be noted that in the case of language teaching, interaction with the agent is in itself an instance of learning practice. one aspect to be studied in the development of educational chatbots is their ability to understand students needs and respond with the style that teachers, trained to educate, use to address their students @xcite . although current llms show great capacity for language generation and for providing relevant -although not always correct or true-answers to different types of queries, it is important to study whether these models can be used in an educational context, being able to respond to a student by simulating a dialogue with a teacher. @xcite propose such an evaluation called the ai teacher test challenge. this paper presents the retuyt-inco submission to the bea 2023 shared task @xcite on generating teacher responses in educational dialogues. in this work, we analyze some particularities of the dataset used in the competition, we describe the approaches we made to solving the problem, and we present the results we obtained, together with an analysis and discussion of future steps. ## data analysis the following study aims to understand the patterns and characteristics of the conversations between teachers and students, which will be crucial for training a chatbot to generate appropriate responses. ## results given that this work is framed in the context of the bea 2023 shared task, and the development and test sets gold responses were not released until after the competition finished, we created our own internal split of the training set in 80% for training and 20% for internal validation. we will present the results of all our experiments against this internal validation data, which we call the internal validation phase. for the development and test sets, we will only present the results of the systems submitted to the competition. a problem with this internal split, as already explained in the data analysis section, is that it includes some repeated utterances across the training and validation sets, due to the overlapping that oc- ## conclusions we presented the experiments we performed for the bea 2023 shared task on generating teacher responses in educational dialogues. our methods use the latest open source llms in a variety of scenarios and incorporating some fine-tuning and targeted prompting strategies for improving the performance. the experiment that yielded best results in the development phase was the model alpaca lora with a few-shot prompting technique, which ranked third. however, in the evaluation phase, the fine-tuning version of opt 2.7b with preprocessing ended up performing better than the previous one, and ranked fourth in this phase.