Supporting Cognitive and Emotional Empathic Writing of Students (2021)

Abstract: We present an annotation approach to capturing emotional and cognitive empathy in student-written peer reviews on business models in German. We propose an annotation scheme that allows us to model emotional and cognitive empathy scores based on three types of review components. Also, we conducted an annotation study with three annotators based on 92 student essays to evaluate our annotation scheme. The obtained inter-rater agreement of α=0.79 for the components and the multi-π=0.41 for the empathy scores indicate that the proposed annotation scheme successfully guides annotators to a substantial to moderate agreement. Moreover, we trained predictive models to detect the annotated empathy structures and embedded them in an adaptive writing support system for students to receive individual empathy feedback independent of an instructor, time, and location. We evaluated our tool in a peer learning exercise with 58 students and found promising results for perceived empathy skill learning, perceived feedback accuracy, and intention to use. Finally, we present our freely available corpus of 500 empathy-annotated, student-written peer reviews on business models and our annotation guidelines to encourage future research on the design and development of empathy support systems.

## introduction empathy is an elementary skill in society for daily interaction and professional communication and is therefore elementary for educational curricula (e.g., learning framework 2030 (oecd, 2018)). it is the "ability to simply understand the other person's perspective @xcite and to react to the ob-figure 1 : empathy annotation scheme. first, a text paragraph is classified into a peer review component (strengths, weakness, improvement suggestions). second, the same annotator is then scoring the cognitive and emotional empathy level of the components based on our annotation guideline on a 1-to-5 scale. to train students with regard to empathy, educational institutions traditionally rely on experiential learning scenarios, such as shadowing, communication skills training, or role playing @xcite . individual empathy training is only available for a limited number of students since individual feedback through a student's learning journey is often hindered due to large-scale lectures or the growing field of distance learning scenarios such as massive open online classes (moocs) @xcite . one possible path for providing individual learning conditions is to leverage recent developments in computational linguistics. language-based models enable the development of writing support systems that provide tailored feedback and recommendations @xcite , e.g., like those already used for argumentation skill learning @xcite . recently, studies have started investigating elaborated models of human emotions (e.g., @xcite , @xcite , @xcite , or @xcite ), but available corpora for empathy detection are still rare. only a few studies address the detection and prediction of empathy in natural texts @xcite , and, to the best of our knowledge, only one corpus is publicly available for empathy modelling based on news story reactions @xcite . past literature therefore lacks 1) publicly available empathy annotated data sets, 2) empathy annotation models based on rigorous annotation guidelines combined with annotation studies to assess the quality of the data, 3) the alignment of empathy in literature on psychological constructs and theories, and 4) an embedding and real-world evaluation of novel modelling approaches in collaborative learning scenarios @xcite . we introduce an empathy annotation scheme and a corpus of 500 student-written reviews that are annotated for the three types of review components, strengths, weaknesses, and suggestions for improvements, and their embedded emotional and cognitive empathy level based on psychological theory @xcite . we trained different models and embedded them as feedback algorithms in a novel writing support tool, which provided students with individual empathy feedback and recommendations in peer learning scenarios. the measured empathy skill learning @xcite , the perceived feedback accuracy @xcite , and the intention to use @xcite in a controlled evaluation with 58 students provided promising results for using our approach in different peer learning scenarios to offer quality education independent of an instructor, time, and location. our contribution is fourfold: 1) we derive a novel annotation scheme for empathy modeling based on psychological theory and previous work on empathy annotation @xcite ; 2) we present an annotation study based on 92 student peer reviews and three annotators to show that the annotation of empathy in student peer reviews is reliably possible; 3) to the best of our knowledge, we present the second freely available corpus for empathy detection in general and the first corpus for empathy detection in the educational domain based on 500 student peer reviews collected in our lecture about business innovation in german; 4) we embedded our annotation approach as predictive models in a writing support system and evaluated it with 58 students in a controlled peer learning scenario. we hope to encourage research on student-written empathetic texts and writing support systems to train students' empathy skills based on nlp towards a quality education independent of a student's location or instructors. ## background the construct of empathy the ability to perceive the feelings of another person and react to their emotions in the right way requires empathy -the ability "of one individual to react to the observed experiences of another" (davis (1983), p.1). empathy plays an essential role in daily life in many practical situations, such as client communication, leadership, or agile teamwork. despite the interdisciplinary research interest, the term empathy is defined from multiple perspectives in terms of its dimensions or components @xcite . aware of the multiple perspectives on empathy, in this annotation study, we focused on the cognitive and emotional components of empathy as defined by davis (1983) and @xcite . therefore, we follow the 'toronto empathy scale' @xcite as a synthesis of instruments for measuring and validating empathy. hence, empathy consists of both emotional and cognitive components @xcite . while emotional empathy lets us perceive what other people feel, cognitive empathy is the human ability to recognize and understand other individuals @xcite . ## corpus construction we compiled a corpus of 500 student-generated peer reviews in which students provided each other with feedback on previously developed business models. peer reviews are a modern learning scenario in large-scale lectures, enabling students to reflect on their content, receive individual feedback from peers, and thus deepen their understanding of the content @xcite . moreover, they are easy to set up in traditional largescale learning scenarios or the growing field of distance-learning scenarios such as moocs. this can be leveraged to train skills such as the ability to appropriately react to other students' perspectives (e.g., santos et al. (2018) ). therefore, we aim to create an annotated corpus to provide empathy feedback based on a data set that a) is based on real-world student peer reviews, b) consists of a sufficient corpus size to be able to train models in a real-world scenario and c) follows a novel annotation guideline for guiding the annotators towards an adequate agreement. hence, we propose a new annotation scheme to model peer review components and their emotional and cognitive empathy levels that reflect the feedback discourse in peer review texts. we base our empathy annotation scheme on emotional and cognitive empathy following davis (1983) and @xcite guided by the study of @xcite . to build a reliable corpus, we followed a 4-step methodology: 1) we examined scientific literature and theory on the construct of empathy and on how to model empathy structures in texts from different domains; 2) we randomly sampled 92 student-generated peer reviews and, on the basis of our findings from literature and theory, developed a set of annotation guidelines consisting of rules and limitations on how to annotate emphatic review discourse structures; 3) we applied, evaluated, and improved our guidelines with three native speakers of german in a total of eight consecutive workshops to resolve annotation ambiguities; 4) we followed the final annotation scheme based on our 14-page guidelines to annotate a corpus of 500 student-generated peer reviews. foot_2 ## conclusion we introduce a novel empathy annotation scheme and an annotated corpus of student-written peer reviews extracted from a real-world learning scenario. our corpus consisted of 500 student-written peer reviews that were annotated for review components and their emotional and cognitive empathy levels. our contribution is threefold: 1) we derived a novel annotation scheme for empathy modeling based on psychological theory and previous work for empathy modeling @xcite ; 2) we present an annotation study based on 92 student peer reviews and three annotators to show that the annotation of empathy in student peer reviews is reliably possible ; and 3) to the best of our knowledge, we present the second freely available corpus for empathy detection and the first corpus for empathy detection in the educational domain based on 500 student peer reviews in german. for future research, this corpus could be leveraged to support students' learning processes, e.g., through a conversational interaction @xcite . however, we would also encourage research on the ethical considerations of empathy detection models in userbased research (i.e., @xcite ). we, therefore, hope to encourage future research on student-generated empathetic texts and on writing support systems to train empathy skills of students based on nlp towards quality education independent of a student's location or instructors. adaptive dialog-based learning system for argumentation skills. in proceedings of the 2021 chi conference on human factors in computing systems. thiemo wambsganss, christina niklaus, matthias cetto, matthias söllner, jan marco leimeister, and siegfried handschuh. 2020a. al : an adaptive learning support system for argumentation skills. in acm chi conference on human factors in computing systems, pages 1-14. thiemo wambsganss, christina niklaus, matthias söllner, siegfried handschuh, and jan marco leimeister. 2020b. a corpus for argumentative writing support in german. in proceedings of the 28th international conference on computational linguistics, pages 856-869, barcelona, spain (online). international committee on computational linguistics. thiemo wambsganss, florian weber, and matthias söllner. 2021c. design and evaluation of an adaptive empathy learning tool. in hawaii international conference on system sciences (hicss). jin wang, liang chih yu, k. robert lai, and xuejie zhang. 2016. dimensional sentiment analysis using a regional cnn-lstm model. 54th annual meeting of the association for computational linguistics, acl 2016 -short papers, pages 225-230. bo xiao, dogan can, panayiotis g georgiou, david atkins, and shrikanth s narayanan. 2012. analyzing the language of therapist empathy in motivational interview based psychotherapy. signal and information processing association annual summit and conference (apsipa), ... asia-pacific. asia-pacific signal and information processing association annual summit and conference, 2012. n. zierau, t wambsganss, andreas janson, sofia schöbel, and jan marco leimeister. 2020. the anatomy of user experience with conversational agents : a taxonomy and propositions of service clues. in international conference on information systems (icis)., pages 1-17.