Don’t Drop the Topic - The Role of the Prompt in Argument Identification in Student Writing (2022)

Abstract: In this paper, we explore the role of topic information in student essays from an argument mining perspective. We cluster a recently released corpus through topic modeling into prompts and train argument identification models on different data settings. Results show that, given the same amount of training data, prompt-specific training performs better than cross-prompt training. However, the advantage can be overcome by introducing large amounts of cross-prompt training data.

## introduction argumentative essays are among the most common essay types students are assigned to write in higher education contexts @xcite . in such an essay, students have to state and justify their opinion on a certain topic elicited by a specific writing prompt. in order to score argumentative essays and give formative feedback automatically, the automatic identification and classification of components in the argumentative structure is important @xcite . while their holistic scoring can be seen as one variant of automatic essay scoring, identifying the argumentative structure within an essay is a natural language processing (nlp) task known as argument mining. argument mining is the automatic identification and extraction of the structure of inference and reasoning expressed as arguments presented in natural language @xcite . the recent kaggle competition "feedback prize -evaluating student writing" 1 can be seen as an argument mining task in an educational scenario, which called on participants to identify argumentative elements in english essays written by u.s. students. figure 1 shows an example from the dataset for an essay where students have been asked to express their 1 https://www.kaggle.com/c/ feedback-prize-2021 attitude towards driverless cars. individual argumentative elements such as position, evidence or concluding statement are highlighted in the text. the argument mining task is not restricted to a certain domain or topic. for example, previous work considered legal @xcite , political @xcite or educational @xcite data. however, it is an open question to what extent argument mining algorithms pick up on topical words indicative not for, e.g., a conclusion in general, but for a conclusion within a specific topic. the feedback prize data mentioned above with its large amount of annotated student essays on various topics offers an ideal opportunity for first steps towards closing this gap. in the data, we notice that very similar sentences can receive different argumentative labels depending on the topic and the context of an essay. for example, the sentence "exercise is really good for your health" was annotated as a claim in an essay on the topic "limiting car usage" while the sentence "(. . . ) running is good for your body" was marked as evidence for the topic "no sports at grade c". such examples highlight the relevance of topic and context information for the argument mining task and give rise to research questions like: to address these questions, we present in this paper experimental studies to investigate the influence of the prompt in an educational argument identifying task using the example of the newly released kaggle feedback prize dataset. ## related work in the following, we discuss related work performing argument mining in the educational domain and work addressing the relevance of topic information. early work treated sentence boundaries as the natural separator of components in an essay. in such a scenario, the identification of argumentative elements boils down to a sentence classification task. for example, @xcite classified sentences as introductory material, position, main/supporting idea, conclusion, title and irrelevant automatically, using features derived from rhetorical structure theory trees and the occurrence of discourse markers. @xcite developed a rule-based algorithm to label each sentence in a student essay into one out of four types (current study, hypothesis, claim, citation). we experimented with sentence classification approaches on the kaggle dataset mentioned above, but found them unsuitable as they do not reflect the gold standard units well. as shown in figure 2 , one sentence can contain multiple argumentative elements, while one argumentative element can span sentences like the lead, counterclaim and evidence annotations in figure 1 . our sentence classification experiments using a support vector machine reached an f1-score of only 0.2. we thus did not further proceed with sentence classification on this dataset. however, driverless cars should be looked at as useful and a positive alternative to everyday cars as they are aware and self-sufficient for their owner's benefit. based on a modification of the toulmin argument model @xcite , @xcite proposed a model of argument components in scientific articles and persuasive essays at the clause-level using four label types -major claim, claim, premise and non-argumentative. their annotation guidelines yielded substantial agreement in an annotation study on 90 persuasive essays in english @xcite . following this schema, the international corpus of learner english @xcite was annotated by @xcite . they trained classification models to identify argument components and used them as features to predict argumentative scores in essays @xcite . in recent research, the granularity of argumentative components was further increased to the token level. in this case, the identification of argumentative elements corresponds to assigning an argument label to each word. in this paradigm, sequence labeling techniques like conditional random fields or pretrained bert models started contributing to argument mining @xcite . the kaggle competition "feedback prize -evaluating student writing" can also be seen as a token labeling task as suggested by the organizers foot_0 . most studies on argument mining mentioned above do not take the topic of the essay into consideration, assuming that arguments can be classified independently of a topic. however, as shown in studies like @xcite , argument mining models did not generalize well on crossdomain data. subsequently, the importance of topic information has drawn more and more attention in the general argument mining task recently: @xcite found that a topic-general model could achieve comparable performance to a topic-specific model by adding limited amounts of topic-specific data. @xcite proved that topic information connected with large pretrained language models like bert provides a significant performance boost in argument mining. however, the effect of topic information has not been fully examined in educational argument mining. the data released by the kaggle competition gives us a chance to investigate this research gap, because it not only contains large amounts of student essays with gold standard annotation of different argumentative elements, but also covers essays from a variety of different writing prompts which, while not being annotated in the dataset, can be automatically inferred. ## data as mentioned before, we use the dataset provided as part of the kaggle competition "feedback prize -evaluating student writing". the dataset consists of 15,594 argumentative students essays written by u.s. students from grades 6 to 12. essays contain annotations for the following argumentative labels: foot_1 argumentative units have been annotated with an overall inter-rater reliability of .73. the lowest reliability was reported for counterclaims and rebuttals (which were often labeled as claims). the highest reliability was found for concluding statements. all disagreements were adjudicated by an expert rater. 4 figure 3 shows the frequency and average number of tokens per span for each label in the dataset. we notice that the argumentative components are very unevenly distributed. claim and evidence occur substantially more frequently than the other labels, with counterclaims and rebuttals being particularly rare. in terms of the length of the underlying span for a label, instances of the types evidence, concluding statement and lead correspond to the longest spans. the average length of all essays is 429 words, while the average length of evidence is 77 words, which means that, given the frequency of the label, evidence is the majority class on the token level. in contrast, position and claim have the shortest average length. ## experimental study 1 -the influence of prompt information in this study, we train argument mining models with different combinations of prompt-specific and cross-prompt data and compare their performance on the same test datasets, in order to investigate our first research question: in how far is argument mining prompt-dependent? furthermore, we analyze the performance difference among argument labels. ## experimental study 2 -what do we actually learn? aiming to answer our second research question of whether the algorithms are more susceptible to prompt-specific or general information, we now transform the original data into topic-only and structure-only versions. ## conclusion this work set out to investigate the importance of topic information in educational argument mining tasks. for this purpose, we first clustered a recently published dataset of student essays into underlying prompts. secondly, we presented a study on the effect of prompt-specific and cross-prompt training material in the identification of argumentative elements. results showed that within-prompt training data is beneficial when a fixed limited amount of training data is used. this advantage can be overcome by larger amounts of additional cross-prompt data. in the analysis of argumentative elements, we found that lead and conclusion can be best identified in all settings, presumably because of their relatively fixed position. lastly, we separated topical from structural information in the essays. from experiments with this modified data, we found that argument mining benefits both from topic words and structure words, i.e. the information is not redundant, but that, unsurprisingly, topical information has a tendency to be more important in within-prompt classification while structure is more relevant across prompts. these findings provide the following insights for future research: first, learning curve studies could investigate an optimal trade-off between topic-specific and generic training data. second, the argumentative elements identified in student essays could be meaningful for the generation of formative feedback directed towards students, such as highlighting different argumentative elements. another research direction is the evaluation of argument quality through analyzing discourse relations between these argument components in order to generate feedback towards coherence and cohesion aspects of student essays.