A Corpus for Argumentative Writing Support in G erman (2020)

Abstract: In this paper, we present a novel annotation approach to capture claims and premises of arguments and their relations in student-written persuasive peer reviews on business models in German language. We propose an annotation scheme based on annotation guidelines that allows to model claims and premises as well as support and attack relations for capturing the structure of argumentative discourse in student-written peer reviews. We conduct an annotation study with three annotators on 50 persuasive essays to evaluate our annotation scheme. The obtained inter-rater agreement of α = 0.57 for argument components and α = 0.49 for argumentative relations indicates that the proposed annotation scheme successfully guides annotators to moderate agreement. Finally, we present our freely available corpus of 1,000 persuasive student-written peer reviews on business models and our annotation guidelines to encourage future research on the design and development of argumentative writing support systems for students.

## introduction in today's world most information is readily available. consequently, the sole reproduction of information is losing attention. this results in a shift of job profiles towards interdisciplinary, ambiguous and creative tasks @xcite . therefore, educational institutions need to evolve in their curricula when it comes to the compositions of skills and knowledge conveyed. in particular, teaching higher order thinking skills to students, such as critical thinking, collaboration or problem-solving, has become more important @xcite . this has already been recognized by the organization for economic co-operation and development (oecd), which included these skills as a major element of their learning framework 2030 @xcite . one subclass represents the skill of arguing in a structured, reflective and well-formed way. argumentation is not only an essential part of our daily communication and thinking, but it also contributes significantly to the competencies of communication, collaboration and problem-solving @xcite . building on studies by aristotle, the ability to form convincing arguments is recognized as the foundation for persuading an audience of novel ideas, and it plays a major role in strategic decision-making and analyzing different standpoints. to develop skills such as argumentation, it is of great importance for the individual student to receive continuous feedback throughout their learning journey, also called formative feedback @xcite . however, this is naturally hindered due to traditional large-scale lectures and due to the growing field of distance learning scenarios such as massive open online courses (moocs) @xcite . in fact, educational institutions, such as universities, face the challenge of providing individual formative feedback effectively @xcite , since every student would need a personal tutor to have an optimal learning environment for learning how to argue @xcite . one possible path for providing individual feedback is to leverage recent developments in computational linguistics in the form of computer-assisted writing which enables the development of writing support systems that provide tailored feedback about textual documents @xcite . argumentation mining (am), a research field in computational linguistics, aims at automatically identifying arguments in unstructured texts @xcite ). an argument is a set of statements made up of three parts: a claim, a set of evidence or premises (e.g., facts) and an inference from the evidence to the claim @xcite . claim and premise represent the argument components. the claim is the central component of an argument, representing an arguable text unit, while the premises are propositions that either support or attack the claim, underpinning its plausibility. support and attack are argumentative relations that model the discourse structure of arguments. in the identification of these argumentation structures, two main tasks can be distinguished: 1) argument component classification, the classification of argumentative text into claims and premises, and 2) argument relation classification, the identification of support and attack relationships between pairs of argument components. to design, train and test am algorithms, the availability of high-quality labeled corpora is crucial. therefore, numerous prior works have dealt with creating annotated data sets. however, they are all limited to a particular genre, ranging from well-structured legal @xcite and scientific documents @xcite , to rather ambiguous, vague and less formal social web content @xcite @xcite . corpora that are applicable for the design and development of argumentative writing support systems are scarce @xcite @xcite . the only collection from the education domain that is annotated for argumentative discourse structures was presented in @xcite . it is composed of 90 persuasive essays written by students in english language and later extended to include 402 essays @xcite . however, these corpora are 1) annotated in english language only and 2) not derived from a specific learning scenario which would leverage the effective use for an argumentative writing support system. consequently, there is a lack of linguistic corpora for training models that provide students with adaptive feedback about the quality of their argumentation in common scenarios in large-scale lectures or the growing field of moocs @xcite , such as peer reviews where students provide each other argumentative feedback on a specific task, e.g., on a previously developed business model @xcite . creating gold standards and test collections requires a formal representation model as well as corresponding annotation guidelines. in this paper, we introduce an argumentation annotation scheme for persuasive student-generated peer reviews extracted from a common learning scenario. moreover, we present a corpus of 1,000 student-written peer reviews that are annotated for argumentation components and their relations. our contribution is threefold: 1) we derive an annotation scheme for a new data domain for am based on argumentation theory and previous work on annotation schemes for persuasive student essays @xcite , 2) we present an annotation study based on 50 persuasive peer reviews and three annotators to show that the annotation of student peer reviews is reliably possible, 3) we present our final and freely available corpus of 1,000 student peer reviews collected in our lecture about business innovation in german language. we therefore hope to encourage future research on student-generated argumentative texts and on writing support systems to train argumentation skills of students based on am. ## corpus construction we compiled a corpus of 1,000 student-generated peer reviews in which students provide each other feedback on previously developed business models. peer reviews are a modern learning scenario in large-scale lectures, enabling students to reflect on their content, receive individual feedback from peers and thus deepen their understanding of the content @xcite . moreover, they are easy to set up in traditional large-scale learning scenarios or in the growing field of distance-learning scenarios such as moocs @xcite . this can be applied to train skills such as argumentation. however, since not many suitable corpora are available to provide argumentation feedback that a) contain annotated persuasive student peer reviews, b) consist of a sufficient size to be able to use trained models in a real-world scenario and c) follow an annotation guideline for guiding the annotators towards an adequate agreement, we propose an new annotation scheme to model argument components as well as argumentation relations that reflects the argumentative discourse structure in persuasive peer reviews. we based our annotation scheme on the model of @xcite and the studies of @xcite . to build a reliable corpus, we followed a four step methodology, illustrated in figure 1 : 1) we examined scientific literature and theory on how to model argumentation discourse structures in texts from different domains, 2) we randomly sampled 50 student-generated peer reviews and, on the basis of our findings from literature and theory, developed a set of annotation guidelines consisting of rules and limitations on how to annotate argumentation discourse structures, 3) we applied, evaluated and improved our guidelines with three native-speakers in three consecutive workshops to resolve annotation ambiguities, 4) we applied the final annotation scheme based on our 15 pages guidelines to a corpus of 1,000 student-generated peer reviews. ## corpus analysis we analyzed the results of the annotation process in order to examine (1) the reliability of the corpus and (2) the major disagreements in argument component and relation annotations between the annotators. in addition, we calculated some statistics of the final corpus. ## application of the corpus after constructing and analysing our corpus, we leveraged the novel data to train a machine learning model. our objective was to embed a classification algorithm in the backend of a user-centered adaptive writing support system to provide students with formative argumentation feedback in the writing process. therefore, we trained and tuned a model with different text features @xcite . we performed a multiclass classification on the sentence level to detect the argument components and their relations. for argument component classification, we found that a support vector machine (svm) achieved the best results, with an accuracy of 65.4% on the test set. regarding the argumentative relation classification, a binary classification task, an svm again achieved the best results on our corpus, obtaining an accuracy of 72.1% on the test set. a detailed description of the features and the embedding of our corpus in a user-centered adaptive writing support tool, as well as its effect on students' argumentation writing skills can be found in @xcite . ## conclusion we introduce an argumentation annotation scheme as well as an annotated corpus of persuasive studentwritten peer reviews extracted from a real-world learning scenario which can be leveraged to provide students feedback on the quality of their argumentation (e.g., by highlighting insufficiently backed claims) through a writing support system. regarding the educational domain, previously presented argument annotation schemes and argumentation corpora fall short on several aspects: they are not derived from a modern learning scenario, they do not follow a systematic methodology based on detailed inter-rater agreement studies or they do not include annotations of argumentative relations. to overcome these limitations, we present a corpus of 1,000 student-written peer reviews that are annotated for argument components and their relations. our contribution is threefold: 1) we derive an annotation scheme for a new data domain for am based on argumentation theory and previous work on annotation schemes for persuasive student essays @xcite , 2) we present an annotation study based on 50 persuasive peer reviews that were annotated by three native german speakers, demonstrating that the annotation of student-generated peer reviews is reliably possible, and 3) we present our final and freely available corpus of 1,000 student peer reviews collected in our lecture about business innovation in german language. we hope to encourage fellow researchers to leverage our annotation scheme and argumentation corpus to design and develop writing support systems for students in large-scale learning scenarios.