Enhancing Educational Dialogues: A Reinforcement Learning Approach for Generating AI Teacher Responses (2023)

Abstract: Reinforcement Learning remains an underutilized method of training and fine-tuning Language Models (LMs) despite recent successes. This paper presents a simple approach of fine-tuning a language model with Reinforcement Learning to achieve competitive performance on the BEA 2023 Shared Task whose goal is to automatically generate teacher responses in educational dialogues. We utilized the novel NLPO algorithm that masks out tokens during generation to direct the model towards generations that maximize a reward function. We show results for both the t5-base model with 220 million parameters from the HuggingFace repository submitted to the leaderboard that, despite its comparatively small size, has achieved a good performance on both test and dev set, as well as GPT-2 with 124 million parameters. The presented results show that despite maximizing only one of the metrics used in the evaluation as a reward function our model scores highly in the other metrics as well.

## introduction controlling the output of language models is a challenging problem in the field of natural language processing (nlp). recently reinforcement learning (rl) has successfully been applied to the training and fine-tuning of language models. chatgpt, based on instructgpt @xcite , makes use of reinforcement learning. @xcite have proposed the grue (general reinforced-language understanding evaluation) benchmark that consists of a variety of different tasks, supervised by different reward functions to measure the quality of the trained models. the reported results on a variety show good results on a variety of tasks. despite recent advances in applying rl to the training and fine-tuning of lms and their wide applicability to different tasks and benchmarks this approach is still not widely applied. in this paper we make use of reinforcement learning-based fine-tuning to tackle the bea 2023 shared task @xcite . the goal of the task is the generation of teacher-like responses in an educational dialogue setting between a student and a teacher. this necessitates that the language model can mimic the tone and overall quality of the teacher response. we have employed an approach that pushes the generations of the model in the right direction through the use of bertscore as a reward function and using reinforcement learning as our training strategy. our model submission to the leaderboard is the implementation of the t5 model @xcite in the huggingface repository, t5-base with 220 million parameters. as the goal is to generate a response given an input dialogue we have chosen a sequence-to-sequence model. we follow the findings of @xcite who suggest that a small model with a high-quality reward function can match or outperform models with magnitudes of more parameters. for the training process we use the dialogue preceding the final teacher response as input and the final teacher response as the reference text. we achieve an average rank across all metrics of 5.38, out of 10 submissions, placing overall in seventh place on the leaderboard. for the dialogrpt maximum weighted ensemble metric our model achieves first place on the test set. we additionally present results for an autoregressive model. the chosen model is the base gpt-2 model from the huggingface repository with 124 million parameters. the autoregressive model outperforms our submitted model despite its smaller size in terms of parameters, suggesting that this model architecture may be more suitable for this task. toledo et al. (2023) demonstrate the viability of a reinforcement learning approach in text-based games. notably they achieve improvements over the previous state of the art in this zero-shot setting. the task of aiding students is comparable due to the large number of possible topics and unforeseen behavior of students when interacting with either a human teacher or a machine teacher. while it is not specifically considered in this task and underrepresented in current research, likely due to the current state of research in this area, there is the possible danger of models becoming outdated in the future, possibly very quickly, as the world around us changes. a solution for this is of course to re-train the models on new data to update them, but a strong performance in a zero-shot setting circumvents this problem altogether, and reinforcement learning approaches show viability in this area. ## data the training data provided for the task by the organizers consists of 2747 samples of student-teacher dialogues from the teacher student chatroom corpus @xcite . there are always two speakers, a student and a teacher, and they take turns talking. each of the samples contains one response. each dialogue turn is prefixed with teacher: or student:, respectively. we use the full input dialogue as the input text, separating each speaker turn by newline. the reference text is the teacher response that follows the input dialogue. we used the t5-base model as well as the gpt2 model from huggingface and their respective tokenizers. to avoid potential issues or the need to cut off samples from the test set we have padded all the in-put tokens to a length of 256 tokens for our model. we note that the task description states that each passage is at most 100 tokens long. the difference in maximum lengths likely comes from our chosen tokenizers, which uses a different tokenizing strategy than the approach that was used to calculate the expected maximum length of 100 tokens. for the training process we used a 80/10/10 split for training-validation-testing of the released training data. ## approach below, we present the methods we developed to generate teacher responses in real-world samples of teacher-student interactions. ## results in ## conclusion in this work we have shown our reinforcement learning-based approach on the bea 2023 shared task. we have used a relatively simple approach and trained two models, t5-base with 220 million parameters and gpt2 with 124 million parameters. despite the overall performance of the models being mixed we have achieved good results in some areas. the gpt-2 model has achieved a good performance on the task and is showing clear gains in terms of evaluation metrics over a zero-shot approach on the same data. this suggests that reinforcement learning-based fine-tuning of language models is a valid approach. according to previous work in the area the model performance when fine-tuned with reinforcement learning is strongly influenced by the quality of the reward function. our approach to this task was very basic and leaves room for improvement, which we believe can be achieved by using both higher quality models instead of relatively small ones with few parameters as well as an improved reward function that makes use of multiple evaluation metrics.