A G erman Corpus of Reflective Sentences (2021)

Abstract: Reflection about a learning process is beneficial to students in higher education (Bub-nys, 2019). The importance of machine understanding of reflective texts grows as applications supporting students become more widespread. Nevertheless, due to the sensitive content, there is no public corpus available yet for the classification of text reflectiveness. We provide the first open-access corpus of reflective student essays in German. We collected essays from three different disciplines (Software Development, Ethics of Artificial Intelligence, and Teacher Training). We annotated the corpus at sentence level with binary reflective/non-reflective labels, using an iterative annotation process with linguistic and didactic specialists, mapping the reflective components found in the data to existing schemes and complementing them. We propose and evaluate linguistic features of reflectiveness and analyse their distribution within the resulted sentences according to their labels. Our contribution constitutes the first open-access corpus to help the community towards a unified approach for reflection detection.

## introduction consciously experienced and reflected practice is a prerequisite for professionalization @xcite . for pre-service teachers, reflection is crucial because it belongs to the core competencies of prospective teachers @xcite @xcite . in literature, several types of reflection can be found. core reflection deals with the core of one's personality: mission and identity @xcite , while self-reflection refers to thinking about one's own behaviour, actions, thoughts or attitudes @xcite . the reflection process can be either guided using prompts to indicate the structure of the reflection @xcite , or free, where educational staff must assess students' reflection texts, yet this is a non-trivial and time-consuming task. machine learning methods can provide possibilities to create such applications. however, the first step towards this is identifying whether reflection is present in a text or not. collections of student essays in machine-readable formats have been created for the last two decades for various machine learning tasks, such as automated essay scoring @xcite , argumentation mining @xcite , reflection detection and automated feedback @xcite . however, to the best of our knowledge, there is no open-source corpus of reflective essays currently available. the reason, in our opinion, lies in the challenges that this kind of data brings. from an ethical point of view, these data are sensitive, since they can be highly personal. in addition, essays are usually collected in an educational setting, and it might be against regulations to publish them. furthermore, inspiring students to reflect is difficult. as a literature review shows, students mostly write descriptive sentences when journaling @xcite . we thus contribute a publicly available, balanced text corpus of reflective and descriptive sentences from students of various universities and disciplines as the first step towards a benchmark for reflection detection in texts. for this, we collected essays from three different sources and anonymized them. we then pre-processed texts into sentences and added manual sentence level annotations according to a synthesised taxonomy, engaging professional linguists and didactic specialists to refine our criteria. we present our quantitative and qualitative linguistic analysis of the resulted corpus. the link to our data can be found in appendix a. ## related work in the context of the multi-genre essay collection, significant works include the british academic written english (bawe) @xcite , the uppsala student english corpus (use) @xcite , and the michigan corpus of upper-level student papers (micusp) @xcite . several efforts were undertaken to create a specialized reflective corpus of students essays at sentence level, namely in pre-service and early teachers settings @xcite or medical students and personnel @xcite . for the didactic case specifically, there has been increasing work in automated detection of reflective sentences in the didactic context @xcite @xcite @xcite @xcite . however, none of the used corpora are publicly available. ## data collection we collected essays of different lengths in both english and german from students and pre-service teachers. we used the sentence segmenter of spacy @xcite to obtain a total of 4232 sentences. during the annotation process, we performed manual anonymization, eliminated all the occurring personal information, including mentioned social media accounts, as well as student and teaching staff names. we describe below how data from the individual sources were collected. for more details on the segmentation, anonymization, and consent processes, see appendix a. dundee teaching placement essays with the agreement of the university of dundee, we scraped 122 reflective essays in english written by students in teacher training during their placements in primary and secondary school in 2018. the students had to upload their essays in the form of an e-portfolio on glow blogs foot_0 , a provider of wordpress tools used by the scottish educational centers. the data reflect their impressions of the scottish educational system in general and school approaches in particular, the acquired skills, their background, role models, insecurities, and motivations to become a teacher. we translated the essays into german using deepl foot_1 and manually corrected conflicting translations that occurred due to inconsistent formatting. after segmentation into simple sentences, we obtained a total of 3595 sentences. ethics of ai and software development using a questionnaire, we collected a set of guided reflective essays in german and english from students of the free university and the technical university of berlin taking a software development project or the ethics of ai lecture. data was collected repeatedly at an interval of a few weeks. the students were asked to reflect on the learning outcome since the previous collection. they were guided by a set of questions developed using gibbs' reflective cycle @xcite , thus spanning the following topics: description of the action they took during their work/learning process, evaluating what they have learned and how to apply it further, what challenges they encountered, and which feelings they note. additionally, they had to rate how their perception and their competencies of the topic changed and to describe why. after segmentation, we obtained a total of 637 sentences. ## annotation process we manually annotated the collected sentences according to the synthesised guidelines presented in section 4. if a sentence met at least one of the enumerated criteria we annotated it as reflective, even if it was a long sentence which also consisted of non-reflective components. the sub-corpora from the software project and ethics of ai lectures were annotated in parallel by four annotators (the first authors and our two collaborating didactic specialists from the friedrich-alexander university erlangen-nürnberg). the initial inter-annotator agreement was low: 0.64 between first authors, 0.32 between first authors and didactic specialists, and 0.33 between the didactic specialists. consequently, we refined our annotation guidelines and re-annotated the dataset. the dundee sub-corpus was annotated by the first author, while the third author annotated 100 random sentences in order to verify consistency. the inter-rater agreement between the annotators was 0.66, which is considered substantial (landis and @xcite . overall, we see that the annotation of reflectiveness is a problematic and tedious task, rather impossible using crowd-sourcing and requiring rounds of discussions and criteria harmonization among inter-disciplinary professionals, as also addressed by @xcite . ## analysis 6.1 methodology we investigate morphological features inspired from @xcite @xcite . however, we hypothesize that reflective sentences also differ in syntactic categories. using a list of respective subordinate conjunctions and punctuation, we extracted main types of subordinate clauses and their length, e.g clause of purpose ('within the framework of our group, we additionally met online on average once a week to share research results and plan the next project steps.', @xmath0; clause of reason ('i volunteered because i want to learn to make better slides and i want to get better at presenting.', @xmath1. we compared the feature distribution in reflective versus non-reflective sentences. the resulted distribution of classes is balanced, with 2177 reflective and 1970 non-reflective sentences. we normalized feature counts according to the number of tokens per sentence, transforming them into frequency counts. as our features were mostly nonnormally distributed, we applied non-parametric u-tests (wilcoxon-mann-whitney) and multipletest correction with benjamini-hochberg procedure (@xmath2. since we find a large number of significant features, we further restricted our criteria. we filtered out features with medians lying on 0 (i.e., where more than 50% of the counts are 0), which is not taken into consideration by the u-test. instead, it considers mean ranks, i.e., the arithmetic average of the positions in the list. ## conclusion with the proposed corpus, we aim to make the first step towards a more unified approach to reflection detection. at the moment, it is not possible to compare existing models, as there is no publicly available benchmark for this task. to address this issue, we created an open-source annotated text corpus of reflective and descriptive sentences from students of various universities and disciplines. we also provide the quantitative and qualitative analysis of the gathered data and describe the annotation procedure and quality assurance measures we took. our work has several limitations. our annotators are not native german speakers, which can influence labeling. however, this will be re-visioned with later versions of the corpus, as we plan to increase the number of annotators and include native speakers. another drawback is the automatic translation of the english data into german. while we plan to quantitatively increase our corpus with german data in the future, the dundee sub-corpus provides a valuable addition. this way, however, it largely influence the results for language-specific features such as subjunctive mood presence, which can appear in translations, but which are still much more common to german than to modern english. we address the low inter-annotators agreement problem with harmonization sessions and refinement of the coding scheme to ensure coverage of complicated instances. we report that with each iteration, inter-annotator agreement increased significantly. thus, we reckon that a fruitful discussion of linguists and specialists of the field in the focus of the task, being a time-consuming process, is the only probable answer to the annotation of cognitive, subjective categories.