A School Student Essay Corpus for Analyzing Interactions of Argumentative Structure and Quality (2024)

Abstract: Learning argumentative writing is challenging. Besides writing fundamentals such as syntax and grammar, learners must select and arrange argument components meaningfully to create high-quality essays. To support argumentative writing computationally, one step is to mine the argumentative structure. When combined with automatic essay scoring, interactions of the argumentative structure and quality scores can be exploited for comprehensive writing support. Although studies have shown the usefulness of using information about the argumentative structure for essay scoring, no argument mining corpus with ground-truth essay quality annotations has been published yet. Moreover, none of the existing corpora contain essays written by school students specifically. To fill this research gap, we present a German corpus of 1,320 essays from school students of two age groups. Each essay has been manually annotated for argumentative structure and quality on multiple levels of granularity. We propose baseline approaches to argument mining and essay scoring, and we analyze interactions between both tasks, thereby laying the ground for quality-oriented argumentative writing support.

## introduction writing argumentative texts, in particular argumentative essays, constitutes an essential part of school students' writing education. however, learning to write arguments of high quality can be challenging @xcite @xcite . it requires various skills, from writing fundamentals, such as syntax and grammar, to argumentation-specific skills, such as meaningfully organizing and structuring arguments and counter-considerations @xcite . this takes time and effort to master @xcite . given teachers' limited time to give students feedback on their writing, automatic argumentative writing support could @xcite dear mrs. grotje, i would prefer to buy sports and play equipment for the school because we want to let off steam during the break and free ourselves from the lessons. we don't want to sit on sofas and talk we sports 2 to 3 times a week, that's the only time we can move around. for the rest of the classes, we mostly sit and listen to the teachers. benefit students as it offers guidance at their own pace and convenience @xcite . argumentative writing support systems employ argument mining to analyze input texts @xcite @xcite , that is, computational methods that identify argumentative components and their relations. common components are major claim (main standpoint of the text, also known as thesis), claim (controversial statement), and premise (reason for justifying or refuting the claim) along with their argumentative relations support and attack. this knowledge enables the systems to give feedback on the structure of a text, e.g., by highlighting unwarranted claims @xcite , or by analyzing the number of argumentative components quantitatively @xcite @xcite . unlike argument mining, automated essay scoring explicitly evaluates essay quality, either holis-tically @xcite @xcite or specific linguistic aspects, such as coherence @xcite , grammar @xcite , and organization @xcite . combining argument mining with essay scoring may enable support systems to give students comprehensive feedback on their writing. in addition, it helps identify how different argumentative structures influence the overall essay quality and which structures are common for different levels of quality @xcite . however, student essay corpora for argument mining are scarce and do not include ground-truth essay quality annotations @xcite . moreover, no corpus with structure annotations for essays written by school students has been published yet. to fill this research gap, we present a german corpus of 1,320 school student essays with manual annotations for argumentative structure and essay quality. the essays have been systematically selected from an existing corpus @xcite , equally distributed over two age groups (fifth-graders and ninth-graders) and binary genders, three per student. we present an extensive annotation scheme focused on school student essays that covers argumentative structure on four levels of granularity as well as five essay quality aspects, as shown in figure 1 . to achieve consistent annotations, we developed annotation guidelines in close dialogue with our expert annotators from the field of language education. this led to high agreement between the annotations. our analyses of the corpus provide various insights into the correlation between the different levels of argumentative structure and essay quality, as well as the interaction between these two types of annotation. we experiment with fine-tuned transformers and adapters as baseline approaches to mining argumentative structure and scoring essay quality. moreover, we demonstrate that the information on argumentative structure helps predicting the essay quality, which is in line with what previous studies showed on other corpora @xcite @xcite . this result underlines the usefulness of our corpus annotations for quality-oriented argumentative writing support. more explicitly, this work aims to answer (i) how the argumentative structure and essay quality of school student essays can be modeled, (ii) how different levels of argumentative structure and essay quality correlate for school student essays, and (iii) how this correlation can be exploited to automatically score the essay quality. altogether, this paper's main contributions are: argumentative writing is a key capability that is taught in school across age groups and disciplines @xcite . a common educational form of argumentative text is the essay, where school students should introduce a thesis, to which they provide pro and con arguments, and finally conclude @xcite . the components of an argumentative text take on different roles (e.g., claim or premises), and they may operationalize different actions (e.g., conceding or reasoning) @xcite . learning to write argumentative text is complex and requires continuous and detailed feedback @xcite . analyzing the argumentative structure of texts computationally, also known as argument(ation) mining, is a crucial and widely-studied step in providing automatic support for argumentative writing @xcite . student essays are a prominent domain for argument mining. a respective annotated corpus of 402 english student essays is available @xcite , for which also quality issues such as insufficient claim support have been modeled @xcite . additionally, student corpora are available for more specific domains, such as argumentative legal texts @xcite , persuasive peer reviews on business models @xcite , and business model pitches @xcite . some research has used argumentative structure to assess essay quality. in consecutive works, @xcite and @xcite @xcite graded different argumentation-related quality aspects for the well-known essay corpus icle @xcite , namely organization, thesis clarity, prompt adherence, and argument strength. in contrast, @xcite targeted different quality aspects of argumentative writing at once. some works further investigated the interaction between argumentative structure and essay quality. @xcite found that multiple argumentation-related essay scoring tasks benefit from argument mining, underlining the impact of argumentative structure on essay quality. the analyses by beigman @xcite and @xcite suggest that this finding also holds for predicting holistic essay scores, while @xcite observed similar for the quality aspect organization. however, these studies relied on automatically assigned quality scores only, due to the lack of ground-truth annotations. in a related line of research, approaches have been proposed to suggest revisions for argumentative essays @xcite , to assess the need for and the quality of revisions @xcite , as well as to perform argument revisions computationally @xcite . other works towards writing support for argumentative essays presented a prototypical system that gives simple feedback in terms of missed criteria @xcite , design principles for an adaptive learning tool (wambsganss and rietsche, 2019), visual feedback to the learner to prompt them to repair broken argument structures @xcite , or point to enthymematic gaps in arguments and make suggestions on how to fill these gaps @xcite . most recently, @xcite proposed a tool that not only analyzes issues with argument quality but also generates an explanation for its prediction. our corpus supports these steps towards support systems for argumentative writing by providing detailed ground-truth annotations for both the argumentative structure and the quality of essays. however, all the works mentioned deal with texts written by university students, while our work targets argumentative essays written by school students, fifth-graders and ninth-graders specifically. to the best of our knowledge, the only other published corpora with school student essays is not openly available @xcite and has been analyzed for essay-level quality aspects only, such as the integration of evidence @xcite and the essay's organization @xcite . we recently came across another school student essay corpus in english with annotations for argumentative structure and quality, which has yet to be published. 2 we go beyond in this work by assessing the quality of school student essays in terms of five aspects derived from language education literature while incorporating their interaction with annotated argumentative structures. this may foster the development of effective methods for helping school students improve their argumentative writing skills. ## school student essay corpus this section presents the source data and annotation of our corpus for analyzing the argumentative structure and quality of school student essays. ## analysis this section reports on our corpus analysis of the interaction between the argumentative structure on micro vs. macro level and component vs. discourse mode level, and the different essay quality aspects. ## experiments this section presents baseline approaches to the two main tasks our corpus enables: predicting the argumentative structure (argument mining) and the essay quality (essay scoring). additionally, we investigate whether information about the argumentative structure helps to predict the essay quality. ## conclusion argumentative writing support of school students presupposes that the quality of their arguments can be assessed. until now, no argument mining corpus with school student essays has been published, let alone any essay corpus with both argument and quality annotations. with this work, we fill both research gaps with a new corpus of 1,320 german school student essays, annotated by experts for argumentative structure and essay quality. our corpus analysis has provided various insights into the correlation between the different levels of argumentative structure and essay quality. in our experiments with fine-tuned transformers and adapters for mining argumentative structure and scoring essay quality we have demonstrated that combining information on all four argumentative structure levels helps the prediction of essay quality. this shows the usefulness of our corpus for research on quality-oriented argumentative writing support, which we seek to enable with this paper. we point out that our corpus contains various information yet to be explored, such as argumentative relations and school student metadata. it thus lays the ground for further analyses-like identifying unwarranted claims and studying differences across age groups and genders.