EDEN : Empathetic Dialogues for E nglish Learning (2024)

Abstract: Dialogue systems have been used as conversation partners in English learning, but few have studied whether these systems improve learning outcomes. Student passion and perseverance, or grit, has been associated with language learning success. Recent work establishes that as students perceive their English teachers to be more supportive, their grit improves. Hypothesizing that the same pattern applies to English-teaching chatbots, we create EDEN, a robust open-domain chatbot for spoken conversation practice that provides empathetic feedback. To construct EDEN, we first train a specialized spoken utterance grammar correction model and a high-quality social chit-chat conversation model. We then conduct a preliminary user study with a variety of strategies for empathetic feedback. Our experiment suggests that using adaptive empathetic feedback leads to higher *perceived affective support*. Furthermore, elements of perceived affective support positively correlate with student grit.

## introduction we study chatbots that teach languages like english @xcite @xcite , and in particular how they can improve student persistence in learning a second language. in the language learning literature, this is referred to as l2 grit @xcite . high l2 grit is crucial for student well-being and success. for example, l2 grit correlates strongly with increased learning enjoyment @xcite , negatively predicts foreign language anxiety and burnout @xcite , and indirectly predicts l2 achievement (khajavy and aghaee, 2022). while l2 grit is often framed as a personality trait, changing an individual's grit is possible @xcite @xcite . as a result, a promising direction is to study grit-improving interventions. figure 1 : the three different empathetic feedback strategies in our experiments. this is a special case where the input is grammatically incorrect, so the no empathetic feedback condition would provide corrections. @xcite examines how teachers' perceived affective support (pas), i.e. how supportive the students perceive their teachers to be @xcite , influences student l2 grit in chinese esl medium-level learners in a college-level english class. the authors discover a strong predictive relationship between higher teacher pas and higher student l2 grit. this further encourages teachers to exhibit warmth and empathy to improve their pas. the study does not establish whether higher teacher pas improves l2 grit, but we posit that it is possible. we seek to determine whether the relationship between pas and l2 grit extends from human teachers to english-teaching dialogue systems. that is, does higher chatbot perceived affective support also increase student l2 grit? since perceived affective support correlates positively with empathy, an empathetic english-teaching chatbot should allow us to study this relationship. however, there is little work on incorporating empathy into open-domain english-teaching chatbots @xcite . to bridge this gap, we construct eden (empathetic dialogues for english learning), which is a high-quality and robust dialogue system capable of empathetic and grammatical feedback. to strengthen components of eden for educational spoken dialogue, we tailor our grammatical feedback for spoken utterances, build a conversation model for open-domain chitchat across multiple topics, and introduce personalization to cater to user preferences. we encourage others to expand upon and customize eden for research within and outside of language education. 1 using eden, we conduct a preliminary user study on how empathetic feedback mechanisms influence the chatbot's perceived affective support and user l2 grit changes. our results suggest that the adaptive empathetic feedback strategy is the most successful in inducing high perceived affective support. this could be due to the specificity of the adaptive mechanism making users feel more thoughtfully attended to. additionally, we discover that certain components of chatbot perceived affective support predict positive changes in l2 grit, which aligns with our hypotheses. ## related work empathetic chatbots have been applied to counseling @xcite , medical assistance @xcite , motivation for weight management @xcite , customer service @xcite , or for social and communicative needs @xcite . however, there is little work on integrating empathy into second-language education using current-day language technologies. there have been affective english educational conversation systems. @xcite , improving upon @xcite , presents a multimodal agent for improving l2 learners' willingness to communicate. the agent carries out a pre-scripted dialogue and adopts different communicative strategies and affective backchannels to reduce learner anxiety. @xcite creates an empathetic spoken chatbot for pronunciation correction using an ontology. @xcite trains a real-world situational chatbot capable of providing feedback. @xcite incorporates persona-based conversation capabilities into a humanoid robot to make language practice easier for anxiety-prone individuals. nonetheless, none of these chatbots account for student emotions. the learner-emotion-aware systems, on the other hand, are often not conversational @xcite @xcite . empathetic strategies have been employed in other forms of learning. affective autotutor (d'mello and graesser, 2013) responds with emotional statements to regulate negative student emotions in physics tutoring. litman and forbes-riley (2014) modifies a spoken physics education system @xcite ) that adapts to user affective states to identify and respond to real-time user disengagement. other affective tutoring systems often offer hints to resolve student frustration @xcite @xcite . we postulate that these approaches may be proven effective for language learning as well. prior work has explored the relationship between chatbot usage and l2 learner experience. @xcite reports korean efl learners experiencing reduced language anxiety and enhanced englishlearning interests when using english chatbots. aimediated discussions have also been more effective than face-to-face discussions for increasing l2 learners' willingness to communicate @xcite . however, there is no systematic study on l2 grit changes in the chatbot context. ## chatbot design while using dialogue systems for english education has become popular, there are only a few fully open-source systems using state-of-the-art methods. in our work, we construct an extensible and robust spoken dialogue system as a conversation partner. figure 2 shows an overview of eden, which makes several key improvements over an empathetic chatbot architecture proposed by @xcite . 2 for each turn, the user input is first analyzed for negative sentiment and prolonged pauses. if these signals are captured, corresponding empathetic feedback is synthesized. otherwise, a grammatical feedback module utilizes a grammar correction model to locate grammar errors and construct feedback using templates. we further devise a grammar correction hierarchy to not overwhelm users (appendix e). when the user utterance does not trigger grammatical feedback, the conversation proceeds as normal through a conversation language model. after the user receives feedback, they occasionally have follow-up queries. we resort to chatgpt for resolving these queries if they are relevant to english learning. we use chatgpt for this purpose rather than our conversation model because responding to user queries about the feedback would be out of the scope of our conversation model. instead, we employ a transition module (appendix h) to continue the original conversation after chatgpt responses. additional design choices are informed by our chatbot design survey conducted on twitter / x with more than 450 responses (appendix b). we make several innovations in our system design to strengthen eden for spoken, educational, and open-domain english-practice dialogue. we discuss how we create a grammar feedback model for spoken utterances (section 3.1) and an opendomain chit-chat model (section 3.2), detail the construction of adaptive empathetic feedback (section 3.3), and specify how users can customize eden according to their needs (section 3.4). ## user study: empathetic feedback we recruited 31 native mandarin speakers from the internet (15) and the authors' home institution (16). the internet participants were not compensated, while the participants from the author's institution received $15 amazon gift cards. our irbapproved study intends to answer these research questions: rq1: does adaptive, empathetic feedback in an english-teaching chatbot result in higher perceived affective support (pas)? rq2: does higher chatbot pas correlate to positive changes in l2 grit? ## conclusion in this work, we build eden, a robust opendomain empathetic english-teaching chatbot tailored for spoken conversations. we then use it to verify whether results from @xcite extend to perceived affective support of chatbots in addition to teachers. our initial user study reveals that higher perceived affective support of our chatbot correlates positively with changes in student l2 grit. we additionally showcase that adaptive, empathetic feedback surpasses fixed and no empathetic feedback in enhancing chatbot perceived affective support. our work serves as a first step in exploring dialogue system interventions for boosting l2 grit.