BIPED : Pedagogically Informed Tutoring System for ESL Education (2024)

Abstract: Large Language Models (LLMs) have a great potential to serve as readily available and cost-efficient Conversational Intelligent Tutoring Systems (CITS) for teaching L2 learners of English. Existing CITS, however, are designed to teach only simple concepts or lack the pedagogical depth necessary to address diverse learning strategies. To develop a more pedagogically informed CITS capable of teaching complex concepts, we construct a BIlingual PEDagogically-informed Tutoring Dataset (BIPED) of one-on-one, human-to-human English tutoring interactions. Through post-hoc analysis of the tutoring interactions, we come up with a lexicon of dialogue acts (34 tutor acts and 9 student acts), which we use to further annotate the collected dataset. Based on a two-step framework of first predicting the appropriate tutor act then generating the corresponding response, we implemented two CITS models using GPT-4 and SOLAR-KO, respectively. We experimentally demonstrate that the implemented models not only replicate the style of human teachers but also employ diverse and contextually appropriate pedagogical strategies.

## introduction as large language models (llms) such as gpt @xcite revolutionize the field of natural language generation, both researchers and practitioners have put an increasing amount of effort into developing conversational intelligent tutoring systems (cits) that leverage the generative capabilities of llm's @xcite @xcite . specifically, llms have the potential to teach english as a second/foreign language (esl/efl), for they may serve as readilyavailable tutors that can emulate native-speaking contexts @xcite ). ## framework to build cits that strategically incorporates pedagogical considerations, we follow the framework @xcite where the llm (1) chooses the appropriate tutor act, (2) then generates the response accordingly. we advocate this two-step approach for the following two reasons. ## method in this section, we introduce two concrete implementations of the framework proposed in section 4. first implementation leverages gpt-4's incontext learning ability and is easily implemented type instruction (1) act prediction ### instruction: select an appropriate next tutor act based on the given 'dialogue context' and 'act candidates'. in the context, tutor uses various tutor act and contents(activities or passages), depending on the student's abilities and questions. tutor act represents a teaching method that you can exploit. choose the best act catering to the context and understanding the meaning of each act." ### input: -context: {dialogue context (appendix g)}, -act candidates: {act candidates (appendix d)} ### response: t.teach.method.vocab.expression.etymology (2) utterance generation ### instruction: as an english tutor for a korean student, your job is to create coherent and tailored responses based on the given 'dialogue context' and 'tutor's act'. in the context, tutor uses various teaching acts and contents(activities or passages), depending on the student's abilities and questions. the given action is a teaching method that you have to exploit. ### input: -context: {dialogue context (appendix g)}, -act: t.teach.method.vocab.expression.etymology, explain a word or a phrase based on its etymology. when possible, discuss the prefix and suffix of the word. ### response: "intro"는 안을 보는 것을 의미하고, "spect"는 "spectrum"할 때 "spect"거든요. "spectrum"은 무언가를 보는 느낌이에요. 그래서 "introspective"는 자기 반성적인 방식으로, 내면을 바라보는 것을 의미해요. (eng) "intro" means looking inward. "spect" is same as "spect" in "spectrum" and feels like seeing something. so, "introspective" means in a self-reflective manner, looking inward. (3) missing context inference ### instruction: predict a speaker and their utterance to fit between the given 'dialogue context' of a student and a teacher and the final 'utterance' made by the teacher. ### input: -context: {partial context (appendix g)}, -utterance: "intro"는 안을 보는 것을 의미하고, "spect"는 "spectrum"할 때 "spect"거든요. "spectrum"은 무언가를 보는 느낌이에요. 그래서 "introspective"는 자기 반성적인 방식으로, 내면을 바라보는 것을 의미해요. (eng) "intro" means looking inward. "spect" is same as "spect" in "spectrum" and feels like seeing something. so, "introspective" means in a self-reflective manner, looking inward. ### response: tutor, 너무 좋아요. (eng) very good. (4) minority act utterance generation ### instruction: as an english teacher for korean students, generate tutor's utterance to teach the given learning content with the act. ### input: -content: [activity 3-9] you likely won't get answers from the other person, but a therapist can help you ___ these issues in an introspective way, durvasula says. (1) vent ( 2 ) navigate (3) abuse (4) withdraw -act: t.teach.method.vocab.expression.etymology, explain a word or a phrase based on its etymology. when possible, discuss the prefix and suffix of the word. ### response: "intro"는 "안쪽의"라는 뜻이고, "spect"는 "보다"라는 뜻입니다. 그래서 "introspective"는 "자아성찰하는"이라는 뜻입니다. (eng) "intro" means "inner" and "spect" means "to see." so, "introspective" means "self-reflective." ## experiments 6.1 dataset split the experiment involved dividing biped randomly into training and test sets, organised by session. this was done based on the distribution of tutors within the dataset, allocating two sessions to one tutor, and one session each to the remaining tutors. biped of 112 tutoring sessions was thus split into 102 training sessions and 10 test sessions. extra data for generating minority act utterances, part of task 4, were only included in the training set. the entire training data was reformulated into different instruction-tuning formats for each task. the test set was devised into 220 scenarios, 10 unique scenarios for every one of the 22 tutor acts in the teaching category. each test scenario was shaped around an act-level utterance initiated by a human tutor applying the respective tutor act, supported by the associated dialogue context and learning content. ## conclusion in this study, we demonstrated the effectiveness of our proposed models in developing conversational intelligent tutoring systems for esl learners. they excelled in replicating the utterance style of human tutors and implementing various and appropriate pedagogical strategies. much of this improvement can be attributed to our newly constructed bilingual and pedagogically-informed tutoring dataset (biped). additionally, our two-step framework and the use of four instruction tuning tasks further contributed to the models' improvement.