Score It All Together: A Multi-Task Learning Study on Automatic Scoring of Argumentative Essays (2023)

Abstract: When scoring argumentative essays in an educational context, not only the presence or absence of certain argumentative elements but also their quality is important. On the recently published student essay dataset PERSUADE, we first show that the automatic scoring of argument quality benefits from additional information about context, writing prompt and argument type. We then explore the different combinations of three tasks: automated span detection, type and quality prediction. Results show that a multi-task learning approach combining the three tasks outperforms sequential approaches that first learn to segment and then predict the quality/type of a segment.

## introduction in educational settings, argumentative essays are considered an important task type, since argumentative writing encourages critical thinking and civic participation @xcite , which can only be developed through practice. while segmentation and classification of argumentative elements in texts are important steps towards providing learners with feedback on their writing structure, assessing the quality of arguments is equally important, but less researched. for instance, in a writing prompt in the recently published essay corpus persuade @xcite , students were asked to write a letter to their principal, who is considering making community service a requirement for all students. two example essays start with the following segments: whether it is cleaning the streets, helping out in an old people's home, or tutoring children at a school, all these deeds are good experience and good for the community. both segments are classified as lead, an argumentative type that should grab the reader's attention and point toward the writer's position. the first segment is labeled as effective, presumably because it captures the reader's interest through listing some activities and effectively shows the author's support for community service. in contrast, likely because it does not point to any position, the second lead has been labeled as ineffective. as humans, we can identify these segments and notice their differences easily, but it is unclear how well a machine can locate, classify and predict the quality rating of each argumentative element. the essay example shown in figure 1 comes from of the kaggle competition dataset "feedback prize -predicting effective arguments" foot_0 , which is a subset of the aforementioned persuade corpus. with the public release of this dataset, largescale argumentative essay data (4192 essays from 15 writing prompts) became publicly available with seven types of argumentative elements being labeled on a three-point quality scale, marking elements as effective, adequate or ineffective. this gives rise to the following research questions: we answer both questions through the following two studies: in study 1, we treat the quality prediction of arguments as a classification task using logistic regression and a large pretrained language model. by including prompt, argumentative type and essay context as additional features, we show that the quality prediction performance benefits from this information. in study 2, we learn the span detection, type and quality prediction tasks in different settings. comparing to a setup where the arguments were first segmented and then classified into different types and quality scales individually, we find that multitask learning settings have a better performance. our source-code is publicly available at https://github.com/yuningding/ multi-task-argument-mining . ## related work with the annotations in the persuade corpus, argument scoring can be seen as a three-step process: the arguments spans are first detected, then classified into different argumentative types and lastly assessed with different quality labels. we structure related work accordingly and begin with work on argument mining, which concerns the first two steps together, before moving to argument quality prediction. finally, we look at the application of multi-task learning in the educational domain. ## data as mentioned above, the data in this paper stems from the kaggle competition "feedback prize -predicting effective arguments", which is a subset of the persuade corpus @xcite . in the competition, the detailed annotation scheme is shared along with an overall inter-annotator agreement (iaa) of .73 on argument types. although the iaa on argument quality labels remains unclear, we manually examined 10% of the essays and were convinced of the validity of the annotation on their effectiveness. we split the data through random sampling. the splitting result is shown in ## study 1 -additional information in argument quality prediction in a first set of experiments, we look at the task of quality prediction in isolation. in other words, we treat the task as a classification task and use gold standard information about span boundaries and -in some conditions -type of the argumentative spans. the objective of this task is to label each span with one out of three labels (effective, adequate, ineffective). in doing so, we want to investigate the influence of three factors, which we assume the classification will benefit from: context, argument type and prompt information. first, we investigate whether argument quality depends on the textual material of the respective argument alone or whether the context of the whole essay is important. we assume that the influence of the essay context might be twofold: first, an argument could make more or less sense depending on its surrounding material. second, we observe that 87% of all essays contain only effective and adequate or only ineffective and adequate arguments. thus, the quality of surrounding elements could help to estimate the quality of an argument. second, we hypothesize that the prediction of argument quality depends on the argument type. a certain argumentative unit could be adequate as lead, but inadequate as conclusion. furthermore, spans of different argument types have different priors to be of a certain quality level. thus, knowing the argument type might help. third, we investigate whether the argument quality prediction is prompt-dependent. the performance of predictions could benefit from prompt information, because students may find certain prompts are more difficult to argument effectively than other prompts. ## conclusion in this paper, three tasks in automatic scoring of argumentative essays were examined, namely argument span detection, type and quality prediction. we found that the quality prediction benefits from prompt information as well as essay context. we further found that, compared to a setup where the arguments were first segmented and then classified into different types and quality scales, multi-task learning settings performed better.